Nov 03 10:25:59.463348 ip-10-0-139-90 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 10:25:59.463358 ip-10-0-139-90 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 10:25:59.463366 ip-10-0-139-90 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 10:25:59.463654 ip-10-0-139-90 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 10:26:09.568400 ip-10-0-139-90 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 10:26:09.568415 ip-10-0-139-90 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b2a24977105942e4ad9735a7c5e7d5aa -- Nov 03 10:28:00.075604 ip-10-0-139-90 systemd[1]: Starting Kubernetes Kubelet... Nov 03 10:28:00.507269 ip-10-0-139-90 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 10:28:00.507269 ip-10-0-139-90 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 10:28:00.507269 ip-10-0-139-90 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 10:28:00.507269 ip-10-0-139-90 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 10:28:00.512628 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512530 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:00.512628 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512625 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:00.512628 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512630 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:00.512628 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512633 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:00.512628 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512636 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:00.512628 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512640 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512643 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512646 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512649 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512652 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512654 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512657 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512660 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512663 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512665 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512668 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512670 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512673 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512675 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512678 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512680 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512683 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512685 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512688 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512690 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:00.512862 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512693 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512697 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512700 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512702 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512705 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512708 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512711 2565 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512721 2565 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512724 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512728 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512733 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512738 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512741 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512744 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512747 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512750 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512754 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512757 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512760 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:00.513350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512763 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512766 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512768 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512771 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512773 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512776 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512778 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512781 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512783 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512786 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512788 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512791 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512794 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512796 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512799 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512801 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512804 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512807 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512810 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:00.513808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512813 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512816 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512819 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512821 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512824 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512827 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512829 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512832 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512835 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512837 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512840 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512842 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512845 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512848 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512850 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512853 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512856 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512858 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512861 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:00.514271 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512863 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512867 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512870 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512873 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512875 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512878 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512880 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512883 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512885 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512888 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512890 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512893 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512897 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512900 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512904 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512906 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512909 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512912 2565 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512915 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512917 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:00.514775 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512920 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512922 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512925 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.512927 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513752 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513760 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513763 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513766 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513769 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513771 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513774 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513777 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513779 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513783 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513786 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513789 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513794 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513797 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513801 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:00.515263 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513805 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513808 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513811 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513814 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513817 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513821 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513824 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513826 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513829 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513832 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513834 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513837 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513840 2565 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513842 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513845 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513848 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513850 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513853 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513856 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513858 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:00.515756 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513861 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513864 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513866 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513869 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513871 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513874 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513876 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513879 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513883 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513885 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513887 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513890 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513893 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513895 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513898 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513900 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513903 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513906 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513909 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:00.516354 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513911 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513914 2565 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513917 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513919 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513923 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513926 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513929 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513932 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513934 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513937 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513939 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513942 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513945 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513947 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513950 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513953 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513955 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513958 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513960 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513963 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:00.516808 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513965 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513968 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513972 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513974 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513977 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513979 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513982 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.513984 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514000 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514003 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514006 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514009 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514012 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514015 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514017 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514020 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514022 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514025 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514027 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514030 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:00.517344 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514033 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514035 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514038 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514041 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514044 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514047 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514049 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514052 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514054 2565 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514057 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514060 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514062 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514602 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514608 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514612 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514615 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514618 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514621 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514623 2565 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514626 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:00.517830 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514629 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514632 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514639 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514644 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514646 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514649 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514652 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514654 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514657 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514660 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514663 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514665 2565 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514668 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514671 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514674 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514676 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514679 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514682 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514684 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514687 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:00.518341 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514689 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514692 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514695 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514697 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514700 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514702 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514705 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514708 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514710 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514713 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514715 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514718 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514721 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514723 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514727 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514732 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514735 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514737 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514740 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:00.518824 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514743 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514745 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514748 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514750 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514753 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514758 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514762 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514765 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514768 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514771 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514774 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514777 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514780 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514782 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514785 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514788 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514790 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514793 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514796 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:00.519286 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514798 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514800 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514804 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514807 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514812 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514815 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514817 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514820 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514823 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514827 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514829 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514832 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514834 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514837 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514839 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514842 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514844 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514847 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514849 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:00.519759 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514852 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514854 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514857 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514859 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514861 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514864 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514867 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514869 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514872 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514874 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514878 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514881 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514883 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514886 2565 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514888 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514890 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514894 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514896 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514899 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514901 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:00.520202 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.514904 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516015 2565 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516026 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516034 2565 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516039 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516044 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516048 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516053 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516058 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516062 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516065 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516069 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516072 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516076 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516079 2565 flags.go:64] FLAG: --cgroup-root="" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516082 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516085 2565 flags.go:64] FLAG: --client-ca-file="" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516088 2565 flags.go:64] FLAG: --cloud-provider="external" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516091 2565 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516096 2565 flags.go:64] FLAG: --cluster-domain="" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516099 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516102 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516105 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516109 2565 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 10:28:00.520693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516113 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516116 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516119 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516123 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516126 2565 flags.go:64] FLAG: --contention-profiling="false" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516130 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516133 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516136 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516139 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516144 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516147 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516150 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516153 2565 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516157 2565 flags.go:64] FLAG: --enable-server="true" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516160 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516165 2565 flags.go:64] FLAG: --event-burst="100" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516169 2565 flags.go:64] FLAG: --event-qps="50" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516172 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516175 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516178 2565 flags.go:64] FLAG: --eviction-hard="" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516182 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516185 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516188 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516191 2565 flags.go:64] FLAG: --eviction-soft="" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516195 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 10:28:00.521269 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516197 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516200 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516203 2565 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516206 2565 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516210 2565 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516213 2565 flags.go:64] FLAG: --feature-gates="" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516217 2565 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516220 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516225 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516228 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516231 2565 flags.go:64] FLAG: --healthz-port="10248" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516234 2565 flags.go:64] FLAG: --help="false" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516238 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516242 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516245 2565 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516248 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516252 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516255 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516258 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516261 2565 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516264 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516267 2565 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516270 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 10:28:00.521961 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516274 2565 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516277 2565 flags.go:64] FLAG: --kube-reserved="" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516280 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516283 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516286 2565 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516288 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516301 2565 flags.go:64] FLAG: --lock-file="" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516304 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516307 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516310 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516316 2565 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516321 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516325 2565 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516333 2565 flags.go:64] FLAG: --logging-format="text" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516336 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516340 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516343 2565 flags.go:64] FLAG: --manifest-url="" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516346 2565 flags.go:64] FLAG: --manifest-url-header="" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516351 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516354 2565 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516358 2565 flags.go:64] FLAG: --max-pods="110" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516361 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516364 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516368 2565 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516372 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 10:28:00.522554 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516375 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516378 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516381 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516389 2565 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516392 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516395 2565 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516399 2565 flags.go:64] FLAG: --pod-cidr="" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516402 2565 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516405 2565 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516408 2565 flags.go:64] FLAG: --pods-per-core="0" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516411 2565 flags.go:64] FLAG: --port="10250" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516414 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516417 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d6fda63dfe66079c" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516420 2565 flags.go:64] FLAG: --qos-reserved="" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516423 2565 flags.go:64] FLAG: --read-only-port="10255" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516427 2565 flags.go:64] FLAG: --register-node="true" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516429 2565 flags.go:64] FLAG: --register-with-taints="" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516434 2565 flags.go:64] FLAG: --registry-burst="10" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516436 2565 flags.go:64] FLAG: --registry-qps="5" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516440 2565 flags.go:64] FLAG: --reserved-cpus="" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516444 2565 flags.go:64] FLAG: --reserved-memory="" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516448 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516451 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516454 2565 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516457 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516460 2565 flags.go:64] FLAG: --runonce="false" Nov 03 10:28:00.523279 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516463 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516466 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516469 2565 flags.go:64] FLAG: --seccomp-default="false" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516472 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516475 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516479 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516483 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516486 2565 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516489 2565 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516492 2565 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516495 2565 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516498 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516501 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516504 2565 flags.go:64] FLAG: --system-cgroups="" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516507 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516513 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516516 2565 flags.go:64] FLAG: --tls-cert-file="" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516519 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516523 2565 flags.go:64] FLAG: --tls-min-version="" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516526 2565 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516529 2565 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516532 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516535 2565 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516538 2565 flags.go:64] FLAG: --v="2" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516542 2565 flags.go:64] FLAG: --version="false" Nov 03 10:28:00.523941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516547 2565 flags.go:64] FLAG: --vmodule="" Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516552 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.516556 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516715 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516721 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516725 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516727 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516731 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516733 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516736 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516739 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516742 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516744 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516752 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516755 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516758 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516760 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516763 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516765 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516768 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:00.524560 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516771 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516773 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516776 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516779 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516781 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516784 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516787 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516789 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516792 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516794 2565 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516797 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516800 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516802 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516806 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516809 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516811 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516815 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516818 2565 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516820 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516823 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:00.525058 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516826 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516829 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516832 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516834 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516837 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516839 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516843 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516846 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516848 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516851 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516854 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516856 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516859 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516862 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516864 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516867 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516870 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516873 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516875 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516878 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:00.525577 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516880 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516883 2565 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516885 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516888 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516891 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516894 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516897 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516899 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516905 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516909 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516912 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516915 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516917 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516920 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516923 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516926 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516928 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516931 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516934 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:00.526056 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516937 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516939 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516942 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516944 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516947 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516950 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516952 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516955 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516958 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516961 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516963 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516966 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516968 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516971 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516974 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516978 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516981 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516984 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516988 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:00.526574 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516991 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516994 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.516998 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517000 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517003 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517005 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517008 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517011 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517013 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517016 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517019 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.517046 2565 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517096 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517102 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517106 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517109 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:00.527049 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517112 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517114 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517117 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517120 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517123 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517125 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517128 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517130 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517133 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517135 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517138 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517141 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517144 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517146 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517149 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517151 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517155 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517158 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517163 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517166 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:00.527465 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517168 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517171 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517174 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517176 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517179 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517181 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517184 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517186 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517189 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517191 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517194 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517197 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517200 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517202 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517205 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517207 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517210 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517212 2565 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517215 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517217 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:00.527947 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517220 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517223 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517225 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517228 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517230 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517233 2565 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517235 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517238 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517242 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517245 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517248 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517251 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517254 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517256 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517259 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517261 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517264 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517266 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517269 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517272 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:00.528455 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517274 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517277 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517279 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517282 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517285 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517287 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517290 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517305 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517308 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517311 2565 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517314 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517316 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517319 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517322 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517324 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517327 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517332 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517336 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517339 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:00.528943 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517342 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517347 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517350 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517355 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517358 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517360 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517363 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517366 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517368 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517371 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517373 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517376 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517379 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517381 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517384 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517387 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517391 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517393 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517396 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:00.529417 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517399 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517402 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517405 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.517408 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.525548 2565 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.525571 2565 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.525591 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.525600 2565 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525674 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525682 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525687 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525692 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525696 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525701 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525705 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525709 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525713 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525718 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525721 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:00.529867 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525725 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525729 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525733 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525738 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525742 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525746 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525751 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525754 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525758 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525762 2565 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525767 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525770 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525774 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525779 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525785 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525791 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525795 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525800 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525806 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:00.530361 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525810 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525814 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525818 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525822 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525826 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525831 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525835 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525840 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525844 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525848 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525852 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525856 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525860 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525864 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525868 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525872 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525876 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525880 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525884 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:00.530856 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525888 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525892 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525896 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525900 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525904 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525908 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525912 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525915 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525919 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525923 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525927 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525932 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525937 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525943 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525947 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525952 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525956 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525959 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525963 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525967 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:00.531332 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525971 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525975 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525980 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525984 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525994 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.525998 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526003 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526007 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526011 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526015 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526019 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526023 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526027 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526031 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526036 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526040 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526044 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526048 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526051 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:00.531922 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526055 2565 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526059 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526064 2565 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526068 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526073 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526077 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526081 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526085 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526090 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526094 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526098 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526104 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526109 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526114 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526118 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526122 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526127 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:00.532446 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526131 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.526165 2565 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526358 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526366 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526370 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526374 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526379 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526382 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526387 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526391 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526395 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526400 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526404 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526408 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526412 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526417 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:00.532900 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526421 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526425 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526429 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526433 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526437 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526442 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526446 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526451 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526455 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526459 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526463 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526467 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526471 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526475 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526480 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526486 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526490 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526495 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526499 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:00.533350 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526502 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526506 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526511 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526537 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526543 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526548 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526552 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526557 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526561 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526565 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526569 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526573 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526577 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526582 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526586 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526590 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526594 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526598 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526602 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526606 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:00.534111 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526610 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526615 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526620 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526624 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526628 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526632 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526635 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526640 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526645 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526649 2565 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526653 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526657 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526661 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526665 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526669 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526673 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526678 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526681 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526685 2565 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526689 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:00.534624 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526693 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526697 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526702 2565 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526706 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526709 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526714 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526717 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526721 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526726 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526730 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526735 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526738 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526742 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526747 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526752 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526757 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526761 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526765 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526769 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:00.535118 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526772 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526776 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526780 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526784 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526788 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526792 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526796 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526801 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526805 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526810 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526814 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526820 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526824 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526829 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.526860 2565 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 10:28:00.535595 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526899 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526905 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526910 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526914 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526917 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526921 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526925 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526929 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526934 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526938 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526942 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526946 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526950 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526955 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526960 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526965 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526970 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526975 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:00.535954 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526979 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526982 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526986 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526990 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526994 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.526998 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527001 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527006 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527010 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527014 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527018 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527022 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527026 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527030 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527034 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527037 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527042 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527046 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527049 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527054 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:00.536631 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527058 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527062 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527066 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527070 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527075 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527079 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527083 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527089 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527095 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527099 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527103 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527107 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527111 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527115 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527119 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527123 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527127 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527130 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527134 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527138 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:00.537401 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527142 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527146 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527150 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527158 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527162 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527166 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527170 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527175 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527179 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527182 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527187 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527190 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527195 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527198 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527202 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527206 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527211 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527215 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527219 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:00.538001 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527224 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527228 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527232 2565 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527236 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527240 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527244 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527248 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527252 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527256 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527260 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527263 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527267 2565 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527271 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527275 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527280 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527284 2565 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527290 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527306 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527310 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527314 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:00.538504 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527319 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527323 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527326 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527331 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527335 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527341 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527345 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527349 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:00.527354 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.527423 2565 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.530278 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.531445 2565 server.go:1014] "Starting client certificate rotation" Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.531549 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.531589 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 10:28:00.538985 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.533887 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 10:28:00.540611 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.540590 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 10:28:00.552252 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.552232 2565 log.go:25] "Validated CRI v1 runtime API" Nov 03 10:28:00.557878 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.557861 2565 log.go:25] "Validated CRI v1 image API" Nov 03 10:28:00.558829 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.558800 2565 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 10:28:00.562337 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.562309 2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bdad2ae6-4a9a-40a6-816f-b33c3fbae43f:/dev/nvme0n1p4 d0ba2c1a-02ed-4fa8-911a-41c7405de004:/dev/nvme0n1p3] Nov 03 10:28:00.562408 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.562336 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 10:28:00.562990 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.562972 2565 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 10:28:00.567853 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.567733 2565 manager.go:217] Machine: {Timestamp:2026-11-03 10:28:00.566222114 +0000 UTC m=+0.386846266 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100328 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217080005f15520b9fb04d257e3093 SystemUUID:ec217080-005f-1552-0b9f-b04d257e3093 BootID:b2a24977-1059-42e4-ad97-35a7c5e7d5aa Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:d2:12:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:d2:12:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:c5:3e:6e:24:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 10:28:00.567949 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.567855 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 10:28:00.567949 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.567943 2565 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 10:28:00.571753 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.571730 2565 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 10:28:00.571955 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.571758 2565 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-90.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 10:28:00.571996 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.571963 2565 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 10:28:00.571996 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.571978 2565 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 10:28:00.572053 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.572016 2565 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 10:28:00.572053 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.572037 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 10:28:00.573264 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.573254 2565 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 10:28:00.577888 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.577876 2565 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 10:28:00.578081 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.578071 2565 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 10:28:00.578121 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.578106 2565 kubelet.go:493] "Attempting to sync node with API server" Nov 03 10:28:00.578121 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.578117 2565 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 10:28:00.578172 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.578128 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 10:28:00.578172 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.578138 2565 kubelet.go:405] "Adding apiserver pod source" Nov 03 10:28:00.578172 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.578148 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 10:28:00.578517 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.578503 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 10:28:00.581045 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.581027 2565 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 10:28:00.589827 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.589807 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 10:28:00.590851 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.590820 2565 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 10:28:00.590950 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.590879 2565 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 10:28:00.591088 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591073 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 10:28:00.591138 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591097 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 10:28:00.591138 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591108 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 10:28:00.591138 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591117 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 10:28:00.591138 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591126 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 10:28:00.591138 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591135 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 10:28:00.591903 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591877 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 10:28:00.591948 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591924 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 10:28:00.591948 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591936 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 10:28:00.591948 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591942 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 10:28:00.592132 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591958 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 10:28:00.592132 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.591969 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 10:28:00.594507 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.594489 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 10:28:00.594599 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.594560 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 10:28:00.598610 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.598594 2565 server.go:1263] "Started kubelet" Nov 03 10:28:00.598696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.598669 2565 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 10:28:00.598788 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.598739 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 10:28:00.598829 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.598807 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 10:28:00.599439 ip-10-0-139-90 systemd[1]: Started Kubernetes Kubelet. Nov 03 10:28:00.600244 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.600220 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7k9fn" Nov 03 10:28:00.600438 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.600348 2565 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 10:28:00.601142 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.601129 2565 server.go:317] "Adding debug handlers to kubelet server" Nov 03 10:28:00.602031 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.602012 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-90.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 10:28:00.602272 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.602244 2565 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 10:28:00.602368 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.602315 2565 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-90.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 10:28:00.605156 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.605132 2565 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 10:28:00.605217 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.605184 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7k9fn" Nov 03 10:28:00.608537 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.608518 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 10:28:00.609775 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.609763 2565 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 10:28:00.609884 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.609867 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610443 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610458 2565 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610476 2565 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610585 2565 reconstruct.go:97] "Volume reconstruction finished" Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610597 2565 reconciler.go:29] "Reconciler: start to sync state" Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610821 2565 factory.go:153] Registering CRI-O factory Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610864 2565 factory.go:223] Registration of the crio container factory successfully Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610923 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610934 2565 factory.go:55] Registering systemd factory Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610943 2565 factory.go:223] Registration of the systemd container factory successfully Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610958 2565 factory.go:103] Registering Raw factory Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.610971 2565 manager.go:1199] Started watching for new ooms in manager Nov 03 10:28:00.614472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.612028 2565 manager.go:319] Starting recovery of all containers Nov 03 10:28:00.615374 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.615320 2565 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:00.620127 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.620102 2565 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-90.us-east-2.compute.internal\" not found" node="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.625695 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.625674 2565 manager.go:324] Recovery completed Nov 03 10:28:00.630734 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.630720 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 10:28:00.632526 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.632508 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 10:28:00.632594 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.632541 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 10:28:00.632594 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.632554 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 10:28:00.633096 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.633082 2565 cpu_manager.go:226] "Starting" policy="none" Nov 03 10:28:00.633096 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.633094 2565 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 10:28:00.633189 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.633087 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 10:28:00.633189 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.633112 2565 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 10:28:00.633736 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.633723 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 10:28:00.633801 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.633748 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 10:28:00.633801 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.633760 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 10:28:00.635146 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.635135 2565 policy_none.go:50] "Start" Nov 03 10:28:00.635185 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.635150 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 10:28:00.635185 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.635161 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 10:28:00.637203 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.637190 2565 policy_none.go:44] "Start" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.681912 2565 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.683213 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.684375 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.684396 2565 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.684418 2565 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.684450 2565 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.686454 2565 manager.go:341] "Starting Device Plugin manager" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.686484 2565 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.686493 2565 server.go:87] "Starting device plugin registration server" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.686766 2565 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.686777 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.686916 2565 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.686977 2565 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.686999 2565 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.687603 2565 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.687651 2565 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:00.689651 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.687662 2565 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:00.784666 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.784577 2565 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"] Nov 03 10:28:00.784666 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.784654 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 10:28:00.785655 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.785640 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 10:28:00.785710 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.785669 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 10:28:00.785710 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.785679 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 10:28:00.787776 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.787761 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 10:28:00.787956 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.787942 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 10:28:00.788079 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788066 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.788123 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788104 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 10:28:00.788671 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788652 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 10:28:00.788789 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788685 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 10:28:00.788789 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788693 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 10:28:00.788789 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788707 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 10:28:00.788789 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788709 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 10:28:00.788789 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788753 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 10:28:00.788789 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788759 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 10:28:00.788789 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788781 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 10:28:00.789053 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788797 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 10:28:00.789053 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.788786 2565 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.790820 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.790803 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.790908 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.790832 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 10:28:00.791547 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.791531 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 10:28:00.791627 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.791556 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 10:28:00.791627 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.791567 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 10:28:00.798289 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.798273 2565 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.798379 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.798308 2565 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-90.us-east-2.compute.internal\": node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:00.811787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.811766 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.811914 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.811791 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.811914 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.811835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.817985 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.817966 2565 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" node="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.820523 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.820506 2565 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" node="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.828991 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.828975 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:00.912136 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.912097 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.912136 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.912134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.912401 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.912178 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.912401 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.912181 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.912401 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.912227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.912401 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:00.912252 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:00.929225 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:00.929200 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:01.030453 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:01.030414 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:01.123524 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.123442 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:01.125810 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.125789 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:01.130876 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:01.130855 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:01.231465 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:01.231426 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:01.332246 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:01.332207 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:01.432936 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:01.432853 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:01.530800 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.530768 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 10:28:01.531301 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.530958 2565 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 10:28:01.531301 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.530969 2565 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 10:28:01.533054 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:01.533028 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:01.611890 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.611824 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 10:23:00 +0000 UTC" deadline="2028-03-29 02:46:28.16946747 +0000 UTC" Nov 03 10:28:01.611890 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.611877 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12280h18m26.557592805s" Nov 03 10:28:01.633348 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:01.633310 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:01.733495 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:01.733424 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:01.786290 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.786251 2565 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:01.810382 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.810349 2565 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:01.823881 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.823850 2565 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 10:28:01.826398 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.826364 2565 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:01.834837 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.834808 2565 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 10:28:01.922063 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:01.922011 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb77e2dfdbc4a16f3bcede022ac475a7.slice/crio-71eff809985da4d6fcb78437170620f5a5f97bb544ab28f0ba74970ef67200c4 WatchSource:0}: Error finding container 71eff809985da4d6fcb78437170620f5a5f97bb544ab28f0ba74970ef67200c4: Status 404 returned error can't find the container with id 71eff809985da4d6fcb78437170620f5a5f97bb544ab28f0ba74970ef67200c4 Nov 03 10:28:01.922743 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:01.922713 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fda1c39c64a6789fa2412aae915cee.slice/crio-a4597f40640b48da624e13186a05078f486220e826d4ed2e16cef82684f15f8a WatchSource:0}: Error finding container a4597f40640b48da624e13186a05078f486220e826d4ed2e16cef82684f15f8a: Status 404 returned error can't find the container with id a4597f40640b48da624e13186a05078f486220e826d4ed2e16cef82684f15f8a Nov 03 10:28:01.929258 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:01.929235 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:28:02.028669 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.028582 2565 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:02.478282 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.478203 2565 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:02.497985 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.497951 2565 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:02.580641 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.580612 2565 apiserver.go:52] "Watching apiserver" Nov 03 10:28:02.591202 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.591170 2565 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 10:28:02.591570 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.591542 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm","openshift-dns/node-resolver-jhp98","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal","openshift-multus/multus-z7d6k","openshift-multus/network-metrics-daemon-ztmr2","openshift-network-diagnostics/network-check-target-t8sqq","openshift-network-operator/iptables-alerter-dgk2v","openshift-ovn-kubernetes/ovnkube-node-fq8sx","kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-4vttz","openshift-cluster-node-tuning-operator/tuned-r877m","openshift-image-registry/node-ca-ssnbk","openshift-multus/multus-additional-cni-plugins-z57kl","kube-system/konnectivity-agent-x9pz8"] Nov 03 10:28:02.594319 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.594292 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.596632 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.596609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.601415 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.599098 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nxhw\"" Nov 03 10:28:02.601415 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.599181 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 10:28:02.601415 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.599443 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.601853 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.601831 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:02.601947 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.601918 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:02.604172 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.604150 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 10:28:02.604542 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.604515 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 10:28:02.604646 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.604550 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 10:28:02.604646 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.604621 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5hs5j\"" Nov 03 10:28:02.604757 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.604668 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 10:28:02.604757 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.604671 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 10:28:02.604757 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.604691 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 10:28:02.604757 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.604712 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 10:28:02.605579 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.605559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:02.605672 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.605647 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:02.605977 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.605957 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 10:28:02.606048 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.605966 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jkrmv\"" Nov 03 10:28:02.608409 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.608284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.608758 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.608739 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 10:28:02.611390 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.611349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.614755 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.614701 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:02.614837 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.614778 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:02.617946 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.617444 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.621475 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621453 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-netns\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-etc-kubernetes\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621748 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-device-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff57n\" (UniqueName: \"kubernetes.io/projected/dcb92e08-27a4-458a-818b-0215acd6dbd5-kube-api-access-ff57n\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-systemd-units\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-slash\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-ovn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-node-log\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-log-socket\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-bin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-system-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.621968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622008 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-socket-dir-parent\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-sys-fs\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-var-lib-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.622293 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-multus\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-conf-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-socket-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-registration-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-systemd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-etc-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35d6f6db-753a-4d1e-ae56-58659a5baab1-hosts-file\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgl8k\" (UniqueName: \"kubernetes.io/projected/35d6f6db-753a-4d1e-ae56-58659a5baab1-kube-api-access-jgl8k\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvrb\" (UniqueName: \"kubernetes.io/projected/068756c4-d0b9-4002-beb5-7661990b1145-kube-api-access-jqvrb\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622716 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-netd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-env-overrides\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovn-node-metrics-cert\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.623195 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d6f6db-753a-4d1e-ae56-58659a5baab1-tmp-dir\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-cnibin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-daemon-config\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622933 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-config\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622965 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-kubelet\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.622995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmgz9\" (UniqueName: \"kubernetes.io/projected/56eddc7b-b1d4-404d-8d08-e482801db81c-kube-api-access-qmgz9\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623021 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-kubelet\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ltn\" (UniqueName: \"kubernetes.io/projected/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-kube-api-access-m8ltn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-hostroot\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-multus-certs\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56eddc7b-b1d4-404d-8d08-e482801db81c-host-slash\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623289 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-bin\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-script-lib\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-k8s-cni-cncf-io\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623397 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-netns\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.624006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-cni-binary-copy\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.624835 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ps6p\" (UniqueName: \"kubernetes.io/projected/de30627e-073b-4b99-9ef3-93dc856e4be6-kube-api-access-4ps6p\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.624835 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623528 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56eddc7b-b1d4-404d-8d08-e482801db81c-iptables-alerter-script\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.624835 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.623555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:02.631552 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.631525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.637749 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.637724 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:02.640436 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.640406 2565 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 10:28:02.640534 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.640518 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wqxzm" Nov 03 10:28:02.650273 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.650217 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wqxzm" Nov 03 10:28:02.691964 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.691911 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"bb77e2dfdbc4a16f3bcede022ac475a7","Type":"ContainerStarted","Data":"71eff809985da4d6fcb78437170620f5a5f97bb544ab28f0ba74970ef67200c4"} Nov 03 10:28:02.694209 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.694157 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerStarted","Data":"a4597f40640b48da624e13186a05078f486220e826d4ed2e16cef82684f15f8a"} Nov 03 10:28:02.708621 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.708593 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 10:28:02.708782 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.708711 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4c7q2\"" Nov 03 10:28:02.709310 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.709285 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-khtn4\"" Nov 03 10:28:02.710106 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710087 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:02.710201 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710136 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 10:28:02.710201 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710188 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 10:28:02.710324 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710316 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:02.710407 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710388 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zcr5r\"" Nov 03 10:28:02.710468 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710451 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 10:28:02.710597 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710581 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wbpkj\"" Nov 03 10:28:02.710706 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710693 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:02.710776 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710762 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 10:28:02.710841 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710806 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 10:28:02.710901 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710856 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-trm8g\"" Nov 03 10:28:02.710958 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710912 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:02.711015 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710972 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xjpbn\"" Nov 03 10:28:02.711015 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.710998 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 10:28:02.711121 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.711053 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 10:28:02.711746 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.711730 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 10:28:02.712150 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.712129 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 10:28:02.712250 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.712162 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 10:28:02.712250 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.712187 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 10:28:02.712250 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.712227 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 10:28:02.712444 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.712252 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 10:28:02.712444 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.712381 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 10:28:02.724706 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.724668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-ovn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.724822 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.724715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovn-node-metrics-cert\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.724822 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.724752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.724822 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.724782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-socket-dir-parent\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.724822 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.724809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.725054 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.724812 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-ovn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.725054 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.724834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-sys\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.725054 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.724884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-conf-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.725219 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725056 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-socket-dir-parent\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.725219 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725095 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-registration-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.725219 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-systemd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.725219 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.725219 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjxv\" (UniqueName: \"kubernetes.io/projected/578bb9eb-99cf-4506-ae47-6d23defa622a-kube-api-access-9zjxv\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35d6f6db-753a-4d1e-ae56-58659a5baab1-hosts-file\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725268 2565 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725271 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725285 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgl8k\" (UniqueName: \"kubernetes.io/projected/35d6f6db-753a-4d1e-ae56-58659a5baab1-kube-api-access-jgl8k\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvrb\" (UniqueName: \"kubernetes.io/projected/068756c4-d0b9-4002-beb5-7661990b1145-kube-api-access-jqvrb\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725322 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-systemd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-netd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725383 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-conf-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725458 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-registration-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.725502 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9bac6548-3f94-4861-84ef-5bf4bfd57409-konnectivity-ca\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-dbus\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725557 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-daemon-config\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysconfig\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-kubernetes\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-tmp\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-kubelet\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-kubelet\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-host\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxzl\" (UniqueName: \"kubernetes.io/projected/c2661da3-c179-4adf-888b-56e2a89b5a02-kube-api-access-2kxzl\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-hostroot\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-multus-certs\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.726006 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.725999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-bin\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-script-lib\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-run\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-host\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-cni-binary-copy\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56eddc7b-b1d4-404d-8d08-e482801db81c-iptables-alerter-script\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726289 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-netns\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-systemd\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-var-lib-kubelet\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-systemd-units\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-log-socket\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-serviceca\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726502 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9bac6548-3f94-4861-84ef-5bf4bfd57409-agent-certs\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-bin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.726787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-system-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726555 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-sys-fs\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726590 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-var-lib-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726606 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-config\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726625 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-multus\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-socket-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-etc-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726856 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-script-lib\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-conf\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-tuned\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726954 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-socket-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-cnibin\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.726990 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-netd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727025 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35d6f6db-753a-4d1e-ae56-58659a5baab1-hosts-file\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.727675 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-env-overrides\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d6f6db-753a-4d1e-ae56-58659a5baab1-tmp-dir\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-cnibin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727138 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-binary-copy\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmgz9\" (UniqueName: \"kubernetes.io/projected/56eddc7b-b1d4-404d-8d08-e482801db81c-kube-api-access-qmgz9\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ltn\" (UniqueName: \"kubernetes.io/projected/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-kube-api-access-m8ltn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727244 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-modprobe-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727287 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-kubelet-config\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56eddc7b-b1d4-404d-8d08-e482801db81c-host-slash\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727396 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgwz\" (UniqueName: \"kubernetes.io/projected/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-kube-api-access-kmgwz\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-k8s-cni-cncf-io\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-netns\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727491 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ps6p\" (UniqueName: \"kubernetes.io/projected/de30627e-073b-4b99-9ef3-93dc856e4be6-kube-api-access-4ps6p\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-node-log\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-lib-modules\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.728551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727652 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-system-cni-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-etc-kubernetes\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727878 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-etc-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-device-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727913 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727943 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-daemon-config\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727953 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-cnibin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff57n\" (UniqueName: \"kubernetes.io/projected/dcb92e08-27a4-458a-818b-0215acd6dbd5-kube-api-access-ff57n\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727979 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-system-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-slash\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728001 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-kubelet\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728003 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728007 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-k8s-cni-cncf-io\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728056 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-multus-certs\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728060 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-bin\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728087 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-hostroot\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728102 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728118 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-slash\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.729412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728136 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-kubelet\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728181 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-bin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728181 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-systemd-units\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728209 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-log-socket\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728248 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728239 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728295 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-var-lib-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728309 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-netns\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.727947 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56eddc7b-b1d4-404d-8d08-e482801db81c-host-slash\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728389 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-multus\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728362 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-sys-fs\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728429 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-netns\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.728462 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728480 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-device-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728522 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-etc-kubernetes\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.728528 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs podName:dcb92e08-27a4-458a-818b-0215acd6dbd5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:03.228506189 +0000 UTC m=+3.049130352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs") pod "network-metrics-daemon-ztmr2" (UID: "dcb92e08-27a4-458a-818b-0215acd6dbd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728638 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-node-log\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.730302 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728647 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-cni-binary-copy\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.731100 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728837 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-config\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.731100 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728934 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-env-overrides\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.731100 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.728962 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d6f6db-753a-4d1e-ae56-58659a5baab1-tmp-dir\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.731100 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.729261 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovn-node-metrics-cert\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.731100 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.729409 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56eddc7b-b1d4-404d-8d08-e482801db81c-iptables-alerter-script\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.733858 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.733830 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 10:28:02.733858 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.733859 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 10:28:02.734028 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.733877 2565 projected.go:196] Error preparing data for projected volume kube-api-access-l7x5g for pod openshift-network-diagnostics/network-check-target-t8sqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:02.734028 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.733935 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g podName:40609c09-1682-4c49-8eae-af9760072fc5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:03.233917327 +0000 UTC m=+3.054541486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l7x5g" (UniqueName: "kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g") pod "network-check-target-t8sqq" (UID: "40609c09-1682-4c49-8eae-af9760072fc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:02.734912 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.734879 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvrb\" (UniqueName: \"kubernetes.io/projected/068756c4-d0b9-4002-beb5-7661990b1145-kube-api-access-jqvrb\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.737516 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.737490 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmgz9\" (UniqueName: \"kubernetes.io/projected/56eddc7b-b1d4-404d-8d08-e482801db81c-kube-api-access-qmgz9\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.737633 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.737553 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ps6p\" (UniqueName: \"kubernetes.io/projected/de30627e-073b-4b99-9ef3-93dc856e4be6-kube-api-access-4ps6p\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.737716 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.737696 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ltn\" (UniqueName: \"kubernetes.io/projected/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-kube-api-access-m8ltn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.738087 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.738047 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgl8k\" (UniqueName: \"kubernetes.io/projected/35d6f6db-753a-4d1e-ae56-58659a5baab1-kube-api-access-jgl8k\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.740155 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.739998 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff57n\" (UniqueName: \"kubernetes.io/projected/dcb92e08-27a4-458a-818b-0215acd6dbd5-kube-api-access-ff57n\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:02.828421 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9bac6548-3f94-4861-84ef-5bf4bfd57409-konnectivity-ca\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:02.828421 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-dbus\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysconfig\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-kubernetes\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-tmp\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828557 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-host\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxzl\" (UniqueName: \"kubernetes.io/projected/c2661da3-c179-4adf-888b-56e2a89b5a02-kube-api-access-2kxzl\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828585 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysconfig\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-run\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828673 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-dbus\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:02.828696 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-host\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-systemd\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828730 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-host\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-var-lib-kubelet\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828778 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828787 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-kubernetes\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-serviceca\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9bac6548-3f94-4861-84ef-5bf4bfd57409-agent-certs\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-conf\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-tuned\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.828963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-cnibin\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-binary-copy\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-modprobe-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-kubelet-config\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgwz\" (UniqueName: \"kubernetes.io/projected/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-kube-api-access-kmgwz\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.829197 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829118 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9bac6548-3f94-4861-84ef-5bf4bfd57409-konnectivity-ca\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.829194 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:02.829297 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret podName:ed900688-4264-4ba8-8daa-83e717a6f13f nodeName:}" failed. No retries permitted until 2026-11-03 10:28:03.329278924 +0000 UTC m=+3.149903071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret") pod "global-pull-secret-syncer-4vttz" (UID: "ed900688-4264-4ba8-8daa-83e717a6f13f") : object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829522 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-conf\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-lib-modules\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829715 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-lib-modules\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829794 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-run\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829803 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-system-cni-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-sys\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829898 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-systemd\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.829970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.829942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjxv\" (UniqueName: \"kubernetes.io/projected/578bb9eb-99cf-4506-ae47-6d23defa622a-kube-api-access-9zjxv\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.830603 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830026 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-modprobe-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.830603 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830080 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-cnibin\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.830603 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830210 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-system-cni-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.830603 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830538 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.830824 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830611 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-var-lib-kubelet\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.830824 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830644 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-binary-copy\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.830824 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830672 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-sys\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.830824 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830683 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.830824 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830720 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-host\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.830824 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830782 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.830824 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830780 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-kubelet-config\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:02.831477 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.830920 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.831477 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.831232 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-serviceca\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.831685 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.831658 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-tmp\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.832091 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.832070 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9bac6548-3f94-4861-84ef-5bf4bfd57409-agent-certs\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:02.832625 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.832594 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-tuned\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.838503 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.838472 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxzl\" (UniqueName: \"kubernetes.io/projected/c2661da3-c179-4adf-888b-56e2a89b5a02-kube-api-access-2kxzl\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.838693 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.838670 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgwz\" (UniqueName: \"kubernetes.io/projected/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-kube-api-access-kmgwz\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.839595 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.839570 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjxv\" (UniqueName: \"kubernetes.io/projected/578bb9eb-99cf-4506-ae47-6d23defa622a-kube-api-access-9zjxv\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:02.916287 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.916003 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:02.920639 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.920598 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z7d6k" Nov 03 10:28:02.930601 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:02.930498 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068756c4_d0b9_4002_beb5_7661990b1145.slice/crio-e31931c322e80c8afa97b10b1cd548b0d128ba65167ac7e1d53d2125ce55169d WatchSource:0}: Error finding container e31931c322e80c8afa97b10b1cd548b0d128ba65167ac7e1d53d2125ce55169d: Status 404 returned error can't find the container with id e31931c322e80c8afa97b10b1cd548b0d128ba65167ac7e1d53d2125ce55169d Nov 03 10:28:02.934662 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:02.934571 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde30627e_073b_4b99_9ef3_93dc856e4be6.slice/crio-da6e4c556c6db14a784e659952326a07ea2ec68109b5372ada31741260f0e2a9 WatchSource:0}: Error finding container da6e4c556c6db14a784e659952326a07ea2ec68109b5372ada31741260f0e2a9: Status 404 returned error can't find the container with id da6e4c556c6db14a784e659952326a07ea2ec68109b5372ada31741260f0e2a9 Nov 03 10:28:02.938501 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.938476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:02.947382 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.947355 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:02.949860 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:02.949764 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d6f6db_753a_4d1e_ae56_58659a5baab1.slice/crio-596e649f74a93fff5772be92eac7e59f0cd8d95345040fce7338e9e66435da4c WatchSource:0}: Error finding container 596e649f74a93fff5772be92eac7e59f0cd8d95345040fce7338e9e66435da4c: Status 404 returned error can't find the container with id 596e649f74a93fff5772be92eac7e59f0cd8d95345040fce7338e9e66435da4c Nov 03 10:28:02.958891 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:02.958862 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56eddc7b_b1d4_404d_8d08_e482801db81c.slice/crio-051e792f9023c91f8519b920f269eb39e1f6a1199e8fdc0c4d1a142f9f9b26c2 WatchSource:0}: Error finding container 051e792f9023c91f8519b920f269eb39e1f6a1199e8fdc0c4d1a142f9f9b26c2: Status 404 returned error can't find the container with id 051e792f9023c91f8519b920f269eb39e1f6a1199e8fdc0c4d1a142f9f9b26c2 Nov 03 10:28:02.975294 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.975266 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:02.980622 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.980589 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:02.981655 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.981627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:02.992277 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:02.992209 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2661da3_c179_4adf_888b_56e2a89b5a02.slice/crio-10f015eec419b99bfb6b7231b6be573f27d8e5225f9fcc725923a234ea010d7f WatchSource:0}: Error finding container 10f015eec419b99bfb6b7231b6be573f27d8e5225f9fcc725923a234ea010d7f: Status 404 returned error can't find the container with id 10f015eec419b99bfb6b7231b6be573f27d8e5225f9fcc725923a234ea010d7f Nov 03 10:28:02.995946 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:02.995919 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f4bb2a_dbbb_4cf1_96bc_e092ffd2efcf.slice/crio-885a845459902eae25f1863fe55ea16efd70b3aae005cd200792d5535322a573 WatchSource:0}: Error finding container 885a845459902eae25f1863fe55ea16efd70b3aae005cd200792d5535322a573: Status 404 returned error can't find the container with id 885a845459902eae25f1863fe55ea16efd70b3aae005cd200792d5535322a573 Nov 03 10:28:02.997493 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:02.997466 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dabb24_0583_4de3_9a5d_21c8ef04c7c5.slice/crio-f05728b61827da09eea34c3b104724b4cd12c729fd64e7332ce5068feeb7315a WatchSource:0}: Error finding container f05728b61827da09eea34c3b104724b4cd12c729fd64e7332ce5068feeb7315a: Status 404 returned error can't find the container with id f05728b61827da09eea34c3b104724b4cd12c729fd64e7332ce5068feeb7315a Nov 03 10:28:02.998816 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:02.998793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:03.006186 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.006015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:03.010243 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:03.010206 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bac6548_3f94_4861_84ef_5bf4bfd57409.slice/crio-b35e39ed7b164fda5ca6fb8cd155503010d6e252f0ad4185ff348c7ba33fd8f7 WatchSource:0}: Error finding container b35e39ed7b164fda5ca6fb8cd155503010d6e252f0ad4185ff348c7ba33fd8f7: Status 404 returned error can't find the container with id b35e39ed7b164fda5ca6fb8cd155503010d6e252f0ad4185ff348c7ba33fd8f7 Nov 03 10:28:03.018456 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:03.018358 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578bb9eb_99cf_4506_ae47_6d23defa622a.slice/crio-0e448fcae41e1f65a439d0cefbd87e6bf0d76f7f785a5a3591fafcf7e0e4bbc4 WatchSource:0}: Error finding container 0e448fcae41e1f65a439d0cefbd87e6bf0d76f7f785a5a3591fafcf7e0e4bbc4: Status 404 returned error can't find the container with id 0e448fcae41e1f65a439d0cefbd87e6bf0d76f7f785a5a3591fafcf7e0e4bbc4 Nov 03 10:28:03.233134 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.233038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:03.233300 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:03.233215 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:03.233300 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:03.233279 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs podName:dcb92e08-27a4-458a-818b-0215acd6dbd5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:04.233265218 +0000 UTC m=+4.053889372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs") pod "network-metrics-daemon-ztmr2" (UID: "dcb92e08-27a4-458a-818b-0215acd6dbd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:03.333488 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.333450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:03.333668 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.333518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:03.333668 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:03.333637 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:03.333668 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:03.333659 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 10:28:03.333860 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:03.333678 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 10:28:03.333860 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:03.333692 2565 projected.go:196] Error preparing data for projected volume kube-api-access-l7x5g for pod openshift-network-diagnostics/network-check-target-t8sqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:03.333860 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:03.333711 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret podName:ed900688-4264-4ba8-8daa-83e717a6f13f nodeName:}" failed. No retries permitted until 2026-11-03 10:28:04.333691034 +0000 UTC m=+4.154315191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret") pod "global-pull-secret-syncer-4vttz" (UID: "ed900688-4264-4ba8-8daa-83e717a6f13f") : object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:03.333860 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:03.333729 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g podName:40609c09-1682-4c49-8eae-af9760072fc5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:04.333717295 +0000 UTC m=+4.154341449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7x5g" (UniqueName: "kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g") pod "network-check-target-t8sqq" (UID: "40609c09-1682-4c49-8eae-af9760072fc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:03.658970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.658914 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 10:23:02 +0000 UTC" deadline="2028-05-07 19:48:24.255802612 +0000 UTC" Nov 03 10:28:03.658970 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.658967 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13233h20m20.596840953s" Nov 03 10:28:03.701472 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.701431 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z57kl" event={"ID":"578bb9eb-99cf-4506-ae47-6d23defa622a","Type":"ContainerStarted","Data":"0e448fcae41e1f65a439d0cefbd87e6bf0d76f7f785a5a3591fafcf7e0e4bbc4"} Nov 03 10:28:03.706561 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.706521 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9pz8" event={"ID":"9bac6548-3f94-4861-84ef-5bf4bfd57409","Type":"ContainerStarted","Data":"b35e39ed7b164fda5ca6fb8cd155503010d6e252f0ad4185ff348c7ba33fd8f7"} Nov 03 10:28:03.708130 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.708096 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ssnbk" event={"ID":"35dabb24-0583-4de3-9a5d-21c8ef04c7c5","Type":"ContainerStarted","Data":"f05728b61827da09eea34c3b104724b4cd12c729fd64e7332ce5068feeb7315a"} Nov 03 10:28:03.711190 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.711157 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r877m" event={"ID":"c2661da3-c179-4adf-888b-56e2a89b5a02","Type":"ContainerStarted","Data":"10f015eec419b99bfb6b7231b6be573f27d8e5225f9fcc725923a234ea010d7f"} Nov 03 10:28:03.712663 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.712637 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jhp98" event={"ID":"35d6f6db-753a-4d1e-ae56-58659a5baab1","Type":"ContainerStarted","Data":"596e649f74a93fff5772be92eac7e59f0cd8d95345040fce7338e9e66435da4c"} Nov 03 10:28:03.715788 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.715759 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z7d6k" event={"ID":"de30627e-073b-4b99-9ef3-93dc856e4be6","Type":"ContainerStarted","Data":"da6e4c556c6db14a784e659952326a07ea2ec68109b5372ada31741260f0e2a9"} Nov 03 10:28:03.720685 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.720538 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerStarted","Data":"885a845459902eae25f1863fe55ea16efd70b3aae005cd200792d5535322a573"} Nov 03 10:28:03.723032 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.723005 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dgk2v" event={"ID":"56eddc7b-b1d4-404d-8d08-e482801db81c","Type":"ContainerStarted","Data":"051e792f9023c91f8519b920f269eb39e1f6a1199e8fdc0c4d1a142f9f9b26c2"} Nov 03 10:28:03.725252 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:03.725223 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" event={"ID":"068756c4-d0b9-4002-beb5-7661990b1145","Type":"ContainerStarted","Data":"e31931c322e80c8afa97b10b1cd548b0d128ba65167ac7e1d53d2125ce55169d"} Nov 03 10:28:04.244017 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.243746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:04.244217 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.243895 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:04.244217 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.244182 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs podName:dcb92e08-27a4-458a-818b-0215acd6dbd5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:06.244162662 +0000 UTC m=+6.064786824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs") pod "network-metrics-daemon-ztmr2" (UID: "dcb92e08-27a4-458a-818b-0215acd6dbd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:04.344849 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.344692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:04.344849 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.344752 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:04.345073 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.344878 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 10:28:04.345073 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.344894 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 10:28:04.345073 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.344904 2565 projected.go:196] Error preparing data for projected volume kube-api-access-l7x5g for pod openshift-network-diagnostics/network-check-target-t8sqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:04.345073 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.344965 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g podName:40609c09-1682-4c49-8eae-af9760072fc5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:06.344935111 +0000 UTC m=+6.165559251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7x5g" (UniqueName: "kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g") pod "network-check-target-t8sqq" (UID: "40609c09-1682-4c49-8eae-af9760072fc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:04.345512 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.345333 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:04.345512 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.345388 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret podName:ed900688-4264-4ba8-8daa-83e717a6f13f nodeName:}" failed. No retries permitted until 2026-11-03 10:28:06.345372907 +0000 UTC m=+6.165997049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret") pod "global-pull-secret-syncer-4vttz" (UID: "ed900688-4264-4ba8-8daa-83e717a6f13f") : object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:04.659305 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.659234 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 10:23:02 +0000 UTC" deadline="2028-04-24 23:08:27.78636659 +0000 UTC" Nov 03 10:28:04.659305 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.659306 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12924h40m23.127077863s" Nov 03 10:28:04.688563 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.687825 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:04.688563 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.687944 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:04.688563 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.688132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:04.688563 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.688224 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:04.688563 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.688282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:04.688563 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:04.688361 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:04.749178 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.749138 2565 generic.go:356] "Generic (PLEG): container finished" podID="29fda1c39c64a6789fa2412aae915cee" containerID="9cf8497ea538d7f17a1b347e73765398c544143d5bf3f8a675a51a5fc1285b2d" exitCode=0 Nov 03 10:28:04.749350 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.749251 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerDied","Data":"9cf8497ea538d7f17a1b347e73765398c544143d5bf3f8a675a51a5fc1285b2d"} Nov 03 10:28:04.751866 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:04.751834 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"bb77e2dfdbc4a16f3bcede022ac475a7","Type":"ContainerStarted","Data":"09d235f1fb0be41d80b42ce7e1ae36c709bfbc1e02e6d43dccbd29f7b60ae16d"} Nov 03 10:28:05.767465 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:05.767422 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerStarted","Data":"a4b32acd22a0544b22d6d9126734e31ffd51bc3aae458560149756074246bfd5"} Nov 03 10:28:05.782569 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:05.782508 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" podStartSLOduration=4.782490622 podStartE2EDuration="4.782490622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:05.782220308 +0000 UTC m=+5.602844468" watchObservedRunningTime="2026-11-03 10:28:05.782490622 +0000 UTC m=+5.603114787" Nov 03 10:28:05.783089 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:05.783052 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" podStartSLOduration=4.783044894 podStartE2EDuration="4.783044894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:04.780226406 +0000 UTC m=+4.600850568" watchObservedRunningTime="2026-11-03 10:28:05.783044894 +0000 UTC m=+5.603669069" Nov 03 10:28:06.259255 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:06.259220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:06.259456 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.259411 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:06.259515 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.259478 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs podName:dcb92e08-27a4-458a-818b-0215acd6dbd5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:10.259457602 +0000 UTC m=+10.080081744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs") pod "network-metrics-daemon-ztmr2" (UID: "dcb92e08-27a4-458a-818b-0215acd6dbd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:06.361818 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:06.361781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:06.361964 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:06.361854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:06.362026 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.361994 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 10:28:06.362026 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.362016 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 10:28:06.362117 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.362034 2565 projected.go:196] Error preparing data for projected volume kube-api-access-l7x5g for pod openshift-network-diagnostics/network-check-target-t8sqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:06.362117 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.362091 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g podName:40609c09-1682-4c49-8eae-af9760072fc5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:10.362073895 +0000 UTC m=+10.182698049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7x5g" (UniqueName: "kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g") pod "network-check-target-t8sqq" (UID: "40609c09-1682-4c49-8eae-af9760072fc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:06.362512 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.362491 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:06.362594 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.362554 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret podName:ed900688-4264-4ba8-8daa-83e717a6f13f nodeName:}" failed. No retries permitted until 2026-11-03 10:28:10.362538764 +0000 UTC m=+10.183162924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret") pod "global-pull-secret-syncer-4vttz" (UID: "ed900688-4264-4ba8-8daa-83e717a6f13f") : object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:06.687486 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:06.687409 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:06.687486 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:06.687448 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:06.687707 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:06.687417 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:06.687707 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.687548 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:06.687707 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.687657 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:06.687838 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:06.687732 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:06.777466 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:06.776165 2565 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="e9f300155bbc73989b028d0818c27d201554c31703b1201be8ca942a5a54b48b" exitCode=0 Nov 03 10:28:06.777466 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:06.777242 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z57kl" event={"ID":"578bb9eb-99cf-4506-ae47-6d23defa622a","Type":"ContainerDied","Data":"e9f300155bbc73989b028d0818c27d201554c31703b1201be8ca942a5a54b48b"} Nov 03 10:28:07.788865 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:07.788826 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9pz8" event={"ID":"9bac6548-3f94-4861-84ef-5bf4bfd57409","Type":"ContainerStarted","Data":"88054c949edb34ad190ffd01a70d40f662063658f76b4a26240902633fe790c3"} Nov 03 10:28:07.807412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:07.806946 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x9pz8" podStartSLOduration=2.894727747 podStartE2EDuration="6.806926863s" totalImagesPullingTime="3.912199116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:07.805167236 +0000 UTC m=+7.625791399" watchObservedRunningTime="2026-11-03 10:28:07.806926863 +0000 UTC m=+7.627551031" Nov 03 10:28:08.685517 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:08.685486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:08.685713 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:08.685666 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:08.685830 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:08.685816 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:08.685914 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:08.685895 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:08.686421 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:08.686404 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:08.686506 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:08.686488 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:08.795876 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:08.795784 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ssnbk" event={"ID":"35dabb24-0583-4de3-9a5d-21c8ef04c7c5","Type":"ContainerStarted","Data":"3520548a80661e430eaf8bc7603c3a374c450205b789d5319e62a727619ad2c5"} Nov 03 10:28:08.800143 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:08.800117 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" event={"ID":"068756c4-d0b9-4002-beb5-7661990b1145","Type":"ContainerStarted","Data":"1d690e011d852cddad464a8dec5c7dcd82759835fdfea29f4c127307c588c070"} Nov 03 10:28:10.297199 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.297167 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:10.297581 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.297371 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:10.297581 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.297433 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs podName:dcb92e08-27a4-458a-818b-0215acd6dbd5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:18.297415782 +0000 UTC m=+18.118039927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs") pod "network-metrics-daemon-ztmr2" (UID: "dcb92e08-27a4-458a-818b-0215acd6dbd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:10.358702 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.357264 2565 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 10:28:10.398577 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.398536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:10.398743 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.398606 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:10.398821 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.398742 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 10:28:10.398821 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.398760 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 10:28:10.398821 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.398773 2565 projected.go:196] Error preparing data for projected volume kube-api-access-l7x5g for pod openshift-network-diagnostics/network-check-target-t8sqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:10.398962 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.398834 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g podName:40609c09-1682-4c49-8eae-af9760072fc5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:18.398815844 +0000 UTC m=+18.219439987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7x5g" (UniqueName: "kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g") pod "network-check-target-t8sqq" (UID: "40609c09-1682-4c49-8eae-af9760072fc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:10.398962 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.398913 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:10.398962 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.398949 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret podName:ed900688-4264-4ba8-8daa-83e717a6f13f nodeName:}" failed. No retries permitted until 2026-11-03 10:28:18.398937724 +0000 UTC m=+18.219561866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret") pod "global-pull-secret-syncer-4vttz" (UID: "ed900688-4264-4ba8-8daa-83e717a6f13f") : object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:10.686451 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.686411 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:10.686603 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.686542 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:10.686964 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.686941 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:10.687093 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.687039 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:10.687155 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.687139 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:10.687242 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:10.687223 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:10.699371 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.698938 2565 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T10:28:10.357279886Z","UUID":"5249a733-e2dd-46ee-9f80-3b384a0756f0","Handler":null,"Name":"","Endpoint":""} Nov 03 10:28:10.704268 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.703925 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 10:28:10.704268 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.703955 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 10:28:10.811503 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.811415 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jhp98" event={"ID":"35d6f6db-753a-4d1e-ae56-58659a5baab1","Type":"ContainerStarted","Data":"5e38c27f295d7282f50ea9a96e06bba679d1dd710a75c7847760c7365e376ea9"} Nov 03 10:28:10.813170 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.813141 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z7d6k" event={"ID":"de30627e-073b-4b99-9ef3-93dc856e4be6","Type":"ContainerStarted","Data":"8ed10825cd443dedff6cc89ad46e337054bcc3a236ec0dc374d3292e9f67ebc5"} Nov 03 10:28:10.816484 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.816457 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" event={"ID":"068756c4-d0b9-4002-beb5-7661990b1145","Type":"ContainerStarted","Data":"eee6a97c31dc1a4a90e792451e03efe39544ffcdafde53393106136ef8e5c591"} Nov 03 10:28:10.826360 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.826312 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ssnbk" podStartSLOduration=5.071769947 podStartE2EDuration="9.826276961s" totalImagesPullingTime="4.754507014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:08.811904925 +0000 UTC m=+8.632529089" watchObservedRunningTime="2026-11-03 10:28:10.826276961 +0000 UTC m=+10.646901123" Nov 03 10:28:10.826643 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.826612 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jhp98" podStartSLOduration=2.304655289 podStartE2EDuration="9.826602258s" totalImagesPullingTime="7.521946969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:10.825575133 +0000 UTC m=+10.646199294" watchObservedRunningTime="2026-11-03 10:28:10.826602258 +0000 UTC m=+10.647226419" Nov 03 10:28:10.842937 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.842894 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-z7d6k" podStartSLOduration=2.780498388 podStartE2EDuration="9.842879771s" totalImagesPullingTime="7.062381383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:10.842426511 +0000 UTC m=+10.663050675" watchObservedRunningTime="2026-11-03 10:28:10.842879771 +0000 UTC m=+10.663503932" Nov 03 10:28:10.848680 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.848655 2565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:10.849236 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:10.849218 2565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:11.821533 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:11.821267 2565 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="ed7ead42173cf7c10d9d35228bb8f11fc08a2252b7727c365edbbe45c4af3db9" exitCode=0 Nov 03 10:28:11.821962 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:11.821824 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z57kl" event={"ID":"578bb9eb-99cf-4506-ae47-6d23defa622a","Type":"ContainerDied","Data":"ed7ead42173cf7c10d9d35228bb8f11fc08a2252b7727c365edbbe45c4af3db9"} Nov 03 10:28:11.823427 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:11.823402 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dgk2v" event={"ID":"56eddc7b-b1d4-404d-8d08-e482801db81c","Type":"ContainerStarted","Data":"f483fb8655636dc1253dca99bee88412799726e3b36cce61b80905da7a4715c4"} Nov 03 10:28:11.827783 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:11.827753 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" event={"ID":"068756c4-d0b9-4002-beb5-7661990b1145","Type":"ContainerStarted","Data":"19bd48bc6ae9721f7ad122c359b10fb1d5c89d23ab6f52a433a7c21dfbc696be"} Nov 03 10:28:11.860168 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:11.860115 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" podStartSLOduration=2.778581207 podStartE2EDuration="10.860097371s" totalImagesPullingTime="8.081516164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:11.859365469 +0000 UTC m=+11.679989629" watchObservedRunningTime="2026-11-03 10:28:11.860097371 +0000 UTC m=+11.680721533" Nov 03 10:28:12.686495 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:12.686451 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:12.686650 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:12.686527 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:12.686650 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:12.686531 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:12.686650 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:12.686636 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:12.686818 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:12.686724 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:12.686818 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:12.686793 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:13.833621 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:13.833582 2565 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="d672a3649217b21958464cb9abfd26e182a502e8ebde05730b44856e97e10762" exitCode=0 Nov 03 10:28:13.834274 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:13.833634 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z57kl" event={"ID":"578bb9eb-99cf-4506-ae47-6d23defa622a","Type":"ContainerDied","Data":"d672a3649217b21958464cb9abfd26e182a502e8ebde05730b44856e97e10762"} Nov 03 10:28:13.856346 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:13.856282 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dgk2v" podStartSLOduration=5.525449232 podStartE2EDuration="12.856268004s" totalImagesPullingTime="7.330818772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:11.873924919 +0000 UTC m=+11.694549084" watchObservedRunningTime="2026-11-03 10:28:13.856268004 +0000 UTC m=+13.676892166" Nov 03 10:28:14.685500 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:14.685443 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:14.685681 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:14.685460 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:14.685681 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:14.685600 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:14.685681 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:14.685473 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:14.685810 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:14.685681 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:14.685810 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:14.685716 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:15.841544 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:15.840922 2565 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="c9e55ce8bd53a868b073b9b25ed501dbe33e783bab43d5329f55439ea684b6f2" exitCode=0 Nov 03 10:28:15.841544 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:15.840977 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z57kl" event={"ID":"578bb9eb-99cf-4506-ae47-6d23defa622a","Type":"ContainerDied","Data":"c9e55ce8bd53a868b073b9b25ed501dbe33e783bab43d5329f55439ea684b6f2"} Nov 03 10:28:16.689349 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:16.689316 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:16.689517 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:16.689356 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:16.689517 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:16.689316 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:16.689517 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:16.689439 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:16.689650 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:16.689520 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:16.689650 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:16.689607 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:17.625002 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.624967 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:17.625501 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.625160 2565 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 10:28:17.625632 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.625611 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:17.848285 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.848118 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:28:17.848668 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.848638 2565 generic.go:356] "Generic (PLEG): container finished" podID="27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf" containerID="ff756edfb62d4f45e84b4f6669014581c405c1d7c8286e6c8231812cd2f39356" exitCode=1 Nov 03 10:28:17.848730 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.848702 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerStarted","Data":"bd55e79bb2af8c3943d0595c6a3a377e4a7c4a8970bc410edd32efa2fede0d69"} Nov 03 10:28:17.848779 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.848768 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerStarted","Data":"bed0e14423b40df4f028b5ce222666caf26b9311d3c5418d71a60579de3848d8"} Nov 03 10:28:17.848825 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.848801 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerStarted","Data":"6afd6a7ca65d8f9ffd74346a18d7f6253fcacdbc83cadb82acf7e615d45837f5"} Nov 03 10:28:17.848856 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.848830 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerStarted","Data":"28aba74ea8f5d1c8230d17243c47038b464bd44988cf3121f5cb137e1c5de8f4"} Nov 03 10:28:17.848891 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.848857 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerDied","Data":"ff756edfb62d4f45e84b4f6669014581c405c1d7c8286e6c8231812cd2f39356"} Nov 03 10:28:17.848920 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:17.848891 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerStarted","Data":"8a6052683e74e641a470f5df45041ee1a35a1af06f24e68ed1da58535ff8e642"} Nov 03 10:28:18.359919 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:18.359831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:18.360089 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.359978 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:18.360089 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.360036 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs podName:dcb92e08-27a4-458a-818b-0215acd6dbd5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:34.360022419 +0000 UTC m=+34.180646562 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs") pod "network-metrics-daemon-ztmr2" (UID: "dcb92e08-27a4-458a-818b-0215acd6dbd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 10:28:18.461162 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:18.461128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:18.461343 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:18.461184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:18.461343 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.461253 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:18.461343 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.461317 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 10:28:18.461343 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.461335 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 10:28:18.461343 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.461347 2565 projected.go:196] Error preparing data for projected volume kube-api-access-l7x5g for pod openshift-network-diagnostics/network-check-target-t8sqq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:18.461594 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.461347 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret podName:ed900688-4264-4ba8-8daa-83e717a6f13f nodeName:}" failed. No retries permitted until 2026-11-03 10:28:34.461326778 +0000 UTC m=+34.281950925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret") pod "global-pull-secret-syncer-4vttz" (UID: "ed900688-4264-4ba8-8daa-83e717a6f13f") : object "kube-system"/"original-pull-secret" not registered Nov 03 10:28:18.461594 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.461422 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g podName:40609c09-1682-4c49-8eae-af9760072fc5 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:34.461398442 +0000 UTC m=+34.282022588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7x5g" (UniqueName: "kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g") pod "network-check-target-t8sqq" (UID: "40609c09-1682-4c49-8eae-af9760072fc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 10:28:18.689468 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:18.689389 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:18.689924 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.689521 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:18.689980 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:18.689926 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:18.690039 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.690022 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:18.690097 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:18.690071 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:18.690149 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:18.690134 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:18.855036 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:18.854999 2565 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="114eb62506484ad1cc984a38ea27de31dafd024844f12f3957446f1739da9f02" exitCode=0 Nov 03 10:28:18.855198 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:18.855072 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z57kl" event={"ID":"578bb9eb-99cf-4506-ae47-6d23defa622a","Type":"ContainerDied","Data":"114eb62506484ad1cc984a38ea27de31dafd024844f12f3957446f1739da9f02"} Nov 03 10:28:19.862594 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:19.862556 2565 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="f93396fdfa90c3007afd4364169a5373ca700d4e6c18b58976ad53864d4fc6b6" exitCode=0 Nov 03 10:28:19.863041 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:19.862670 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z57kl" event={"ID":"578bb9eb-99cf-4506-ae47-6d23defa622a","Type":"ContainerDied","Data":"f93396fdfa90c3007afd4364169a5373ca700d4e6c18b58976ad53864d4fc6b6"} Nov 03 10:28:19.868238 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:19.868129 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:28:19.868540 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:19.868516 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerStarted","Data":"f44dce7950cd9783c1cd8c1510c86c57ae96e354a57b6d94f383d587d2f2e0f3"} Nov 03 10:28:20.688187 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:20.688153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:20.688362 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:20.688266 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:20.688362 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:20.688275 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:20.688362 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:20.688290 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:20.688501 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:20.688365 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:20.688501 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:20.688426 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:20.876119 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:20.876004 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z57kl" event={"ID":"578bb9eb-99cf-4506-ae47-6d23defa622a","Type":"ContainerStarted","Data":"3a283a20b19e375a20ff06236d259197142f6af58d8ff3344a4bc953527a4010"} Nov 03 10:28:20.877527 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:20.877503 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r877m" event={"ID":"c2661da3-c179-4adf-888b-56e2a89b5a02","Type":"ContainerStarted","Data":"d980b29724a5071385b2a163a3d781bf4a0bbb6e38a2be88d5e1c3db65a768f9"} Nov 03 10:28:20.899876 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:20.899828 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z57kl" podStartSLOduration=8.609841242 podStartE2EDuration="19.89981508s" totalImagesPullingTime="11.289973838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:20.899206674 +0000 UTC m=+20.719830834" watchObservedRunningTime="2026-11-03 10:28:20.89981508 +0000 UTC m=+20.720439241" Nov 03 10:28:20.914973 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:20.914930 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r877m" podStartSLOduration=2.751334538 podStartE2EDuration="19.914916411s" totalImagesPullingTime="17.163581873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:20.914582699 +0000 UTC m=+20.735206861" watchObservedRunningTime="2026-11-03 10:28:20.914916411 +0000 UTC m=+20.735540618" Nov 03 10:28:22.688439 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:22.688242 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:22.689166 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:22.688287 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:22.689166 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:22.688521 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:22.689166 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:22.688310 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:22.689166 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:22.688583 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:22.689166 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:22.688648 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:22.885729 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:22.885702 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:28:22.886083 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:22.886059 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerStarted","Data":"3de5960cbd7d954019b658884f2b9bd1df606d859533109aaddc8fcfddc57179"} Nov 03 10:28:22.886455 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:22.886430 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:22.886585 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:22.886544 2565 scope.go:122] "RemoveContainer" containerID="ff756edfb62d4f45e84b4f6669014581c405c1d7c8286e6c8231812cd2f39356" Nov 03 10:28:22.902914 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:22.902895 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:23.554577 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.554545 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ztmr2"] Nov 03 10:28:23.554801 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.554728 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:23.554873 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:23.554855 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:23.558973 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.558940 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8sqq"] Nov 03 10:28:23.559106 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.559062 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:23.559192 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:23.559169 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:23.559553 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.559532 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4vttz"] Nov 03 10:28:23.559650 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.559630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:23.559739 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:23.559722 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:23.891538 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.891465 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:28:23.891967 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.891818 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" event={"ID":"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf","Type":"ContainerStarted","Data":"e92573a270632d0aa9a36e23dda49359a9883000c2c56a2e05a06d2b64335e4f"} Nov 03 10:28:23.892077 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.892046 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:23.892221 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.892128 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:23.907760 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.907725 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:23.917934 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:23.917882 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" podStartSLOduration=8.776520799 podStartE2EDuration="22.917833618s" totalImagesPullingTime="14.141312819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:23.916069028 +0000 UTC m=+23.736693186" watchObservedRunningTime="2026-11-03 10:28:23.917833618 +0000 UTC m=+23.738457775" Nov 03 10:28:25.685289 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:25.685254 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:25.685812 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:25.685254 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:25.685812 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:25.685374 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:25.685812 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:25.685395 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:25.685812 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:25.685457 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:25.685812 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:25.685518 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:27.684722 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:27.684686 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:27.685260 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:27.684692 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:27.685260 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:27.684795 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ztmr2" podUID="dcb92e08-27a4-458a-818b-0215acd6dbd5" Nov 03 10:28:27.685260 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:27.684696 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:27.685260 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:27.684865 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8sqq" podUID="40609c09-1682-4c49-8eae-af9760072fc5" Nov 03 10:28:27.685260 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:27.684946 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vttz" podUID="ed900688-4264-4ba8-8daa-83e717a6f13f" Nov 03 10:28:29.520075 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.520032 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeReady" Nov 03 10:28:29.520477 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.520170 2565 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 10:28:29.554399 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.554368 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-h65vg"] Nov 03 10:28:29.581701 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.581658 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8rp5q"] Nov 03 10:28:29.589076 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.586443 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:29.589380 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.589356 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs"] Nov 03 10:28:29.589495 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.589476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.603739 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.600154 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 10:28:29.603739 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.600259 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 10:28:29.603739 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.600319 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.603739 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.600378 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.603739 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.600443 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 10:28:29.603739 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.600527 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fb6wm\"" Nov 03 10:28:29.603739 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.600564 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 10:28:29.603739 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.601839 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xm5kr\"" Nov 03 10:28:29.604435 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.604411 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 10:28:29.606877 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.606851 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-678cf4bb67-v4rz9"] Nov 03 10:28:29.607018 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.606994 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:29.625490 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.625469 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2m2xr"] Nov 03 10:28:29.625613 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.625585 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.639144 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.639123 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q22k4"] Nov 03 10:28:29.639243 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.639226 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.663636 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.663571 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f"] Nov 03 10:28:29.663742 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.663682 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q22k4" Nov 03 10:28:29.676319 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.676287 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-788447d7cc-tl5jd"] Nov 03 10:28:29.676438 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.676387 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.687245 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.687226 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h"] Nov 03 10:28:29.687422 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.687352 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.696589 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.696568 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.696589 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.696575 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.696750 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.696626 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 10:28:29.696750 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.696635 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.696855 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.696844 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 10:28:29.696855 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.696849 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.697097 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.697082 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 10:28:29.697176 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.697082 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.697176 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.697113 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.697176 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.697114 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.697176 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.697132 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8kqg8\"" Nov 03 10:28:29.697360 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.697290 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.698274 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698259 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 10:28:29.698274 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698265 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 10:28:29.698411 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698284 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 10:28:29.698411 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698309 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 10:28:29.698411 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698318 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2smdw\"" Nov 03 10:28:29.698411 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698341 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.698637 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698574 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 10:28:29.698637 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698576 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 10:28:29.698637 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698583 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.698637 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698599 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 10:28:29.698637 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698601 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m29gn\"" Nov 03 10:28:29.698637 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698624 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 10:28:29.698913 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.698813 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7khc5\"" Nov 03 10:28:29.700110 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.700092 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-58tzx\"" Nov 03 10:28:29.702569 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.702552 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wk475"] Nov 03 10:28:29.702689 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.702676 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:29.702881 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.702867 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 10:28:29.704177 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.704161 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 10:28:29.709072 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.709053 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 10:28:29.710151 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.710129 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 10:28:29.716713 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.716693 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk"] Nov 03 10:28:29.716812 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.716795 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:29.716872 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.716808 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:29.716872 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.716810 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:29.717116 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.717094 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wk475" Nov 03 10:28:29.730204 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730184 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5a9404d9-1240-451f-8064-255583bb5da6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:29.730316 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.730316 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.730316 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzh9\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.730441 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-service-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.730441 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:29.730503 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:29.730532 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730502 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-config\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.730621 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730603 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.730656 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.730688 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730659 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.730688 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.730749 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-serving-cert\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.730749 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.730811 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730753 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/5a9404d9-1240-451f-8064-255583bb5da6-kube-api-access-2pd85\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:29.730853 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.730906 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aedffb88-1606-439b-968d-ab12c5a2b984-serving-cert\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.730956 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.730945 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.731126 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.731108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-tmp\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.731192 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.731141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:29.731192 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.731169 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8j4x\" (UniqueName: \"kubernetes.io/projected/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-kube-api-access-m8j4x\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.731356 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.731199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgkf\" (UniqueName: \"kubernetes.io/projected/69dd95c4-b504-46d3-97a9-ac864aef3f85-kube-api-access-wbgkf\") pod \"kas-connection-checker-685895c7c9-q22k4\" (UID: \"69dd95c4-b504-46d3-97a9-ac864aef3f85\") " pod="kube-system/kas-connection-checker-685895c7c9-q22k4" Nov 03 10:28:29.731356 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.731239 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-snapshots\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.731356 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.731265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpxw\" (UniqueName: \"kubernetes.io/projected/aedffb88-1606-439b-968d-ab12c5a2b984-kube-api-access-vlpxw\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.736960 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.736942 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw"] Nov 03 10:28:29.737058 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.737046 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.750358 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.750340 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qkwqb"] Nov 03 10:28:29.750460 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.750447 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" Nov 03 10:28:29.753037 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.753010 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-675dcb4c7d-4p5mj"] Nov 03 10:28:29.753124 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.753111 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" Nov 03 10:28:29.757748 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.757729 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf"] Nov 03 10:28:29.757855 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.757840 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.769626 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.769609 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-h65vg"] Nov 03 10:28:29.769708 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.769637 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q22k4"] Nov 03 10:28:29.769708 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.769658 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8rp5q"] Nov 03 10:28:29.769708 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.769680 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs"] Nov 03 10:28:29.769708 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.769701 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jhr4s"] Nov 03 10:28:29.769837 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.769721 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" Nov 03 10:28:29.793869 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.793843 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fb26v"] Nov 03 10:28:29.793996 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.793964 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:29.796652 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.796634 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.796771 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.796756 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.796939 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.796918 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.796998 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.796933 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 10:28:29.796998 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.796978 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 10:28:29.796998 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.796987 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.796998 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.796994 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.797159 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797000 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 10:28:29.797225 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797208 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.797270 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797217 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.797270 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797263 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 10:28:29.797551 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797535 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 10:28:29.797612 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797542 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 10:28:29.797612 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797554 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.797612 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797580 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pthxs\"" Nov 03 10:28:29.797612 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797584 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mh9pr\"" Nov 03 10:28:29.797745 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797614 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lq8vs\"" Nov 03 10:28:29.797745 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.797616 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ssvsh\"" Nov 03 10:28:29.798421 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.798397 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z6b7k\"" Nov 03 10:28:29.798421 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.798426 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fhcv2\"" Nov 03 10:28:29.798597 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.798452 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k5ll2\"" Nov 03 10:28:29.798597 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.798469 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 10:28:29.798597 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.798494 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 10:28:29.798597 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.798540 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 10:28:29.798734 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.798608 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w9nrb\"" Nov 03 10:28:29.817998 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.817978 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678cf4bb67-v4rz9"] Nov 03 10:28:29.817998 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818008 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-788447d7cc-tl5jd"] Nov 03 10:28:29.818163 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818029 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw"] Nov 03 10:28:29.818163 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818052 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk"] Nov 03 10:28:29.818163 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818075 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qkwqb"] Nov 03 10:28:29.818163 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818095 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675dcb4c7d-4p5mj"] Nov 03 10:28:29.818163 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818114 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2m2xr"] Nov 03 10:28:29.818163 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818131 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:29.818418 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818134 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h"] Nov 03 10:28:29.818418 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818312 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fb26v"] Nov 03 10:28:29.818418 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818347 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhr4s"] Nov 03 10:28:29.818418 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818375 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f"] Nov 03 10:28:29.818418 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818399 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf"] Nov 03 10:28:29.818584 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.818423 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wk475"] Nov 03 10:28:29.832131 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.832248 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsx67\" (UniqueName: \"kubernetes.io/projected/0dbdd6e7-a8de-4990-9aec-358480af8521-kube-api-access-dsx67\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.832248 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832169 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.832248 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzh9\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.832420 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-service-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.832420 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832280 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:29.832420 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.832313 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:29.832420 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.832329 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678cf4bb67-v4rz9: secret "image-registry-tls" not found Nov 03 10:28:29.832420 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832328 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.832420 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.832374 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls podName:369e7292-b645-4eb6-9a5d-62e08910f1ef nodeName:}" failed. No retries permitted until 2026-11-03 10:28:30.332360556 +0000 UTC m=+30.152984717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls") pod "image-registry-678cf4bb67-v4rz9" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef") : secret "image-registry-tls" not found Nov 03 10:28:29.832420 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmsxs\" (UniqueName: \"kubernetes.io/projected/e86d4545-6769-4133-ba67-674e17ab3392-kube-api-access-xmsxs\") pod \"network-check-source-6f78d59f99-wkrtf\" (UID: \"e86d4545-6769-4133-ba67-674e17ab3392\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" Nov 03 10:28:29.832420 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.832401 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:29.832420 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832420 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffflv\" (UniqueName: \"kubernetes.io/projected/3e7c1aa6-a7c5-4ed6-a871-43a9bf568693-kube-api-access-ffflv\") pod \"kas-connection-checker-685895c7c9-wk475\" (UID: \"3e7c1aa6-a7c5-4ed6-a871-43a9bf568693\") " pod="kube-system/kas-connection-checker-685895c7c9-wk475" Nov 03 10:28:29.832823 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.832466 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls podName:5a9404d9-1240-451f-8064-255583bb5da6 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:30.332453228 +0000 UTC m=+30.153077372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rchbs" (UID: "5a9404d9-1240-451f-8064-255583bb5da6") : secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:29.832952 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.832925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.833126 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8bf626d-af0f-47d8-8d61-686c390139af-serving-cert\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.833214 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833167 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.833276 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.833357 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833277 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-default-certificate\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.833357 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8j4x\" (UniqueName: \"kubernetes.io/projected/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-kube-api-access-m8j4x\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.833357 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833343 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-service-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.833556 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6gx\" (UniqueName: \"kubernetes.io/projected/7e537c04-9b2a-4d85-9ffd-db08c554c838-kube-api-access-ss6gx\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:29.833556 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833406 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.833556 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgkf\" (UniqueName: \"kubernetes.io/projected/69dd95c4-b504-46d3-97a9-ac864aef3f85-kube-api-access-wbgkf\") pod \"kas-connection-checker-685895c7c9-q22k4\" (UID: \"69dd95c4-b504-46d3-97a9-ac864aef3f85\") " pod="kube-system/kas-connection-checker-685895c7c9-q22k4" Nov 03 10:28:29.833760 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833727 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5a9404d9-1240-451f-8064-255583bb5da6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:29.833847 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.833847 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.833941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833844 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-config\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.833941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833920 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.834026 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.833981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qwf\" (UniqueName: \"kubernetes.io/projected/b86c6b47-2b14-4151-a04a-12c1505e928a-kube-api-access-47qwf\") pod \"volume-data-source-validator-58c9f5f46d-h8sdw\" (UID: \"b86c6b47-2b14-4151-a04a-12c1505e928a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" Nov 03 10:28:29.834100 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:29.834150 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/5a9404d9-1240-451f-8064-255583bb5da6-kube-api-access-2pd85\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:29.834203 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5bzs\" (UniqueName: \"kubernetes.io/projected/29923b63-cb6e-4a44-b098-ebd0317c1c14-kube-api-access-w5bzs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.834282 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.834366 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.834422 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.834467 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:29.834670 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7gc\" (UniqueName: \"kubernetes.io/projected/e8bf626d-af0f-47d8-8d61-686c390139af-kube-api-access-xs7gc\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.834750 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834697 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.834840 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-serving-cert\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.834911 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.834911 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.835027 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aedffb88-1606-439b-968d-ab12c5a2b984-serving-cert\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.835027 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834923 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.835027 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.834945 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8pjb\" (UniqueName: \"kubernetes.io/projected/1dd4ba4e-edba-4732-9f72-54c83b78d6fc-kube-api-access-s8pjb\") pod \"kas-connection-checker-685895c7c9-qkwqb\" (UID: \"1dd4ba4e-edba-4732-9f72-54c83b78d6fc\") " pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" Nov 03 10:28:29.835237 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835050 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.835237 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835206 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5a9404d9-1240-451f-8064-255583bb5da6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:29.835370 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835348 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:29.835568 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:29.835635 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.835635 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835594 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-config\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.835742 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-tmp\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.835742 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-snapshots\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.835742 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpxw\" (UniqueName: \"kubernetes.io/projected/aedffb88-1606-439b-968d-ab12c5a2b984-kube-api-access-vlpxw\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.835949 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29923b63-cb6e-4a44-b098-ebd0317c1c14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.836047 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.835969 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qk2c\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.836047 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.836005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:29.836187 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.836049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29923b63-cb6e-4a44-b098-ebd0317c1c14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.836187 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.836122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.836187 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.836155 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-stats-auth\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.836393 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.836190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8bf626d-af0f-47d8-8d61-686c390139af-config\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.836573 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.836554 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-tmp\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.836622 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.836564 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.837407 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.836726 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 10:28:29.837407 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.836727 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.837407 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.836836 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert podName:cb00dfc2-12f0-4fc3-ba37-0bc83b344602 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:30.336795738 +0000 UTC m=+30.157419891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h65vg" (UID: "cb00dfc2-12f0-4fc3-ba37-0bc83b344602") : secret "networking-console-plugin-cert" not found Nov 03 10:28:29.837407 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.836973 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-snapshots\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.838324 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.837751 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.840406 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.840368 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.840619 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.840598 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-serving-cert\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.841596 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.841567 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.841760 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.841737 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.842332 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.842017 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aedffb88-1606-439b-968d-ab12c5a2b984-serving-cert\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.843458 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.843429 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8j4x\" (UniqueName: \"kubernetes.io/projected/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-kube-api-access-m8j4x\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.843869 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.843836 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgkf\" (UniqueName: \"kubernetes.io/projected/69dd95c4-b504-46d3-97a9-ac864aef3f85-kube-api-access-wbgkf\") pod \"kas-connection-checker-685895c7c9-q22k4\" (UID: \"69dd95c4-b504-46d3-97a9-ac864aef3f85\") " pod="kube-system/kas-connection-checker-685895c7c9-q22k4" Nov 03 10:28:29.844228 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.844210 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzh9\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:29.844434 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.844413 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpxw\" (UniqueName: \"kubernetes.io/projected/aedffb88-1606-439b-968d-ab12c5a2b984-kube-api-access-vlpxw\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.845214 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.845182 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/5a9404d9-1240-451f-8064-255583bb5da6-kube-api-access-2pd85\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:29.896860 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.896822 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 10:28:29.897027 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.896901 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 10:28:29.897027 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.896913 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6w9j8\"" Nov 03 10:28:29.934783 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.934709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:29.937039 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.937096 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmsxs\" (UniqueName: \"kubernetes.io/projected/e86d4545-6769-4133-ba67-674e17ab3392-kube-api-access-xmsxs\") pod \"network-check-source-6f78d59f99-wkrtf\" (UID: \"e86d4545-6769-4133-ba67-674e17ab3392\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" Nov 03 10:28:29.937096 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffflv\" (UniqueName: \"kubernetes.io/projected/3e7c1aa6-a7c5-4ed6-a871-43a9bf568693-kube-api-access-ffflv\") pod \"kas-connection-checker-685895c7c9-wk475\" (UID: \"3e7c1aa6-a7c5-4ed6-a871-43a9bf568693\") " pod="kube-system/kas-connection-checker-685895c7c9-wk475" Nov 03 10:28:29.937203 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8bf626d-af0f-47d8-8d61-686c390139af-serving-cert\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.937203 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.937180 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 10:28:29.937311 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.937246 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:30.437227829 +0000 UTC m=+30.257852002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : secret "router-metrics-certs-default" not found Nov 03 10:28:29.937311 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-cert\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:29.937419 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-default-certificate\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.937419 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6gx\" (UniqueName: \"kubernetes.io/projected/7e537c04-9b2a-4d85-9ffd-db08c554c838-kube-api-access-ss6gx\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:29.937510 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmls2\" (UniqueName: \"kubernetes.io/projected/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-kube-api-access-jmls2\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:29.937510 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.937604 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-config-volume\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:29.937604 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:29.937749 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-tmp-dir\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:29.937749 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.937749 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qwf\" (UniqueName: \"kubernetes.io/projected/b86c6b47-2b14-4151-a04a-12c1505e928a-kube-api-access-47qwf\") pod \"volume-data-source-validator-58c9f5f46d-h8sdw\" (UID: \"b86c6b47-2b14-4151-a04a-12c1505e928a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" Nov 03 10:28:29.937749 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz8vn\" (UniqueName: \"kubernetes.io/projected/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-kube-api-access-wz8vn\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:29.937941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937778 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bzs\" (UniqueName: \"kubernetes.io/projected/29923b63-cb6e-4a44-b098-ebd0317c1c14-kube-api-access-w5bzs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.937941 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.937804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.937941 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.937911 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:29.937941 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.937926 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675dcb4c7d-4p5mj: secret "image-registry-tls" not found Nov 03 10:28:29.938149 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.938132 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls podName:c634e6ef-e187-4409-a0a0-bae5793bc137 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:30.438111982 +0000 UTC m=+30.258736128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls") pod "image-registry-675dcb4c7d-4p5mj" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137") : secret "image-registry-tls" not found Nov 03 10:28:29.938216 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:29.938216 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938131 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.938216 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7gc\" (UniqueName: \"kubernetes.io/projected/e8bf626d-af0f-47d8-8d61-686c390139af-kube-api-access-xs7gc\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.938382 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.938240 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 10:28:29.938382 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.938382 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.938322 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls podName:7e537c04-9b2a-4d85-9ffd-db08c554c838 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:30.438276441 +0000 UTC m=+30.258900582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69g9h" (UID: "7e537c04-9b2a-4d85-9ffd-db08c554c838") : secret "samples-operator-tls" not found Nov 03 10:28:29.938382 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.938582 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8pjb\" (UniqueName: \"kubernetes.io/projected/1dd4ba4e-edba-4732-9f72-54c83b78d6fc-kube-api-access-s8pjb\") pod \"kas-connection-checker-685895c7c9-qkwqb\" (UID: \"1dd4ba4e-edba-4732-9f72-54c83b78d6fc\") " pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" Nov 03 10:28:29.938582 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:29.938582 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.938582 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29923b63-cb6e-4a44-b098-ebd0317c1c14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.938582 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qk2c\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.938582 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938559 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29923b63-cb6e-4a44-b098-ebd0317c1c14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.939004 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.939004 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-stats-auth\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.939004 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8bf626d-af0f-47d8-8d61-686c390139af-config\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.939004 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938697 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsx67\" (UniqueName: \"kubernetes.io/projected/0dbdd6e7-a8de-4990-9aec-358480af8521-kube-api-access-dsx67\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.939004 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.938722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.939326 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.939156 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.939326 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.939176 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:29.940189 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.939642 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.940189 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.939742 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29923b63-cb6e-4a44-b098-ebd0317c1c14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.940189 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:29.939954 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:30.439937145 +0000 UTC m=+30.260561300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : configmap references non-existent config key: service-ca.crt Nov 03 10:28:29.940189 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.940158 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8bf626d-af0f-47d8-8d61-686c390139af-config\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.940467 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.940341 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-default-certificate\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.940467 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.940346 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8bf626d-af0f-47d8-8d61-686c390139af-serving-cert\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.940778 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.940758 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.941016 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.940994 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.941540 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.941520 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-stats-auth\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.941678 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.941664 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29923b63-cb6e-4a44-b098-ebd0317c1c14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.946591 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.946550 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qwf\" (UniqueName: \"kubernetes.io/projected/b86c6b47-2b14-4151-a04a-12c1505e928a-kube-api-access-47qwf\") pod \"volume-data-source-validator-58c9f5f46d-h8sdw\" (UID: \"b86c6b47-2b14-4151-a04a-12c1505e928a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" Nov 03 10:28:29.947218 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.947192 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6gx\" (UniqueName: \"kubernetes.io/projected/7e537c04-9b2a-4d85-9ffd-db08c554c838-kube-api-access-ss6gx\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:29.949573 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.949147 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.949573 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.949241 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bzs\" (UniqueName: \"kubernetes.io/projected/29923b63-cb6e-4a44-b098-ebd0317c1c14-kube-api-access-w5bzs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:29.949573 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.949468 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsx67\" (UniqueName: \"kubernetes.io/projected/0dbdd6e7-a8de-4990-9aec-358480af8521-kube-api-access-dsx67\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:29.949573 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.949499 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7gc\" (UniqueName: \"kubernetes.io/projected/e8bf626d-af0f-47d8-8d61-686c390139af-kube-api-access-xs7gc\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:29.949787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.949577 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8pjb\" (UniqueName: \"kubernetes.io/projected/1dd4ba4e-edba-4732-9f72-54c83b78d6fc-kube-api-access-s8pjb\") pod \"kas-connection-checker-685895c7c9-qkwqb\" (UID: \"1dd4ba4e-edba-4732-9f72-54c83b78d6fc\") " pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" Nov 03 10:28:29.949787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.949671 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffflv\" (UniqueName: \"kubernetes.io/projected/3e7c1aa6-a7c5-4ed6-a871-43a9bf568693-kube-api-access-ffflv\") pod \"kas-connection-checker-685895c7c9-wk475\" (UID: \"3e7c1aa6-a7c5-4ed6-a871-43a9bf568693\") " pod="kube-system/kas-connection-checker-685895c7c9-wk475" Nov 03 10:28:29.949787 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.949772 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmsxs\" (UniqueName: \"kubernetes.io/projected/e86d4545-6769-4133-ba67-674e17ab3392-kube-api-access-xmsxs\") pod \"network-check-source-6f78d59f99-wkrtf\" (UID: \"e86d4545-6769-4133-ba67-674e17ab3392\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" Nov 03 10:28:29.949892 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.949875 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qk2c\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:29.970187 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.970171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:29.989720 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:29.989702 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q22k4" Nov 03 10:28:30.013770 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.013744 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.040060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmls2\" (UniqueName: \"kubernetes.io/projected/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-kube-api-access-jmls2\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.040108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-config-volume\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.040129 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.040145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-tmp-dir\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.040172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz8vn\" (UniqueName: \"kubernetes.io/projected/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-kube-api-access-wz8vn\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.040651 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-tmp-dir\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.040752 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-cert\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.041279 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.041921 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls podName:f8ea16e9-1545-4881-b7c2-883d8b09a6fe nodeName:}" failed. No retries permitted until 2026-11-03 10:28:30.541900742 +0000 UTC m=+30.362524902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls") pod "dns-default-fb26v" (UID: "f8ea16e9-1545-4881-b7c2-883d8b09a6fe") : secret "dns-default-metrics-tls" not found Nov 03 10:28:30.042464 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.041829 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-config-volume\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:30.043488 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.043453 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-cert\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:30.049471 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.049396 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz8vn\" (UniqueName: \"kubernetes.io/projected/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-kube-api-access-wz8vn\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:30.050111 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.050057 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmls2\" (UniqueName: \"kubernetes.io/projected/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-kube-api-access-jmls2\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:30.087033 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.087004 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wk475" Nov 03 10:28:30.108642 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.108614 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:30.132930 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.132905 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" Nov 03 10:28:30.171599 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.171569 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" Nov 03 10:28:30.227656 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.227209 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" Nov 03 10:28:30.243752 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.243491 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:30.301324 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.300944 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2m2xr"] Nov 03 10:28:30.309484 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.306229 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q22k4"] Nov 03 10:28:30.309484 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.308831 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8rp5q"] Nov 03 10:28:30.313794 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:30.313752 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7b33fa_69cf_4582_a4b4_844ed91e2e26.slice/crio-53b6aeee6f808d6fb6ea455381b1fbb98c8f2fd329357c9c440415b319b9688c WatchSource:0}: Error finding container 53b6aeee6f808d6fb6ea455381b1fbb98c8f2fd329357c9c440415b319b9688c: Status 404 returned error can't find the container with id 53b6aeee6f808d6fb6ea455381b1fbb98c8f2fd329357c9c440415b319b9688c Nov 03 10:28:30.316162 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:30.316117 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dd95c4_b504_46d3_97a9_ac864aef3f85.slice/crio-95765fea9b25353b77dc1b85ad2509a2b7987d01d6407e960614d2a1972fb0d3 WatchSource:0}: Error finding container 95765fea9b25353b77dc1b85ad2509a2b7987d01d6407e960614d2a1972fb0d3: Status 404 returned error can't find the container with id 95765fea9b25353b77dc1b85ad2509a2b7987d01d6407e960614d2a1972fb0d3 Nov 03 10:28:30.343634 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.343548 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f"] Nov 03 10:28:30.343796 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.343774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:30.343884 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.343843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:30.343964 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.343879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:30.344732 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.344063 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:30.344732 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.344119 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls podName:5a9404d9-1240-451f-8064-255583bb5da6 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:31.344104848 +0000 UTC m=+31.164729000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rchbs" (UID: "5a9404d9-1240-451f-8064-255583bb5da6") : secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:30.344732 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.344473 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 10:28:30.344732 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.344513 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert podName:cb00dfc2-12f0-4fc3-ba37-0bc83b344602 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:31.344501987 +0000 UTC m=+31.165126140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h65vg" (UID: "cb00dfc2-12f0-4fc3-ba37-0bc83b344602") : secret "networking-console-plugin-cert" not found Nov 03 10:28:30.344732 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.344554 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:30.344732 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.344561 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678cf4bb67-v4rz9: secret "image-registry-tls" not found Nov 03 10:28:30.344732 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.344583 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls podName:369e7292-b645-4eb6-9a5d-62e08910f1ef nodeName:}" failed. No retries permitted until 2026-11-03 10:28:31.34457458 +0000 UTC m=+31.165198733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls") pod "image-registry-678cf4bb67-v4rz9" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef") : secret "image-registry-tls" not found Nov 03 10:28:30.350257 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:30.350221 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29923b63_cb6e_4a44_b098_ebd0317c1c14.slice/crio-2da30282695c66f194a8923a7abcdf27b0a6f3ee2028ed120ae5f6bc04141445 WatchSource:0}: Error finding container 2da30282695c66f194a8923a7abcdf27b0a6f3ee2028ed120ae5f6bc04141445: Status 404 returned error can't find the container with id 2da30282695c66f194a8923a7abcdf27b0a6f3ee2028ed120ae5f6bc04141445 Nov 03 10:28:30.389495 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.389416 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wk475"] Nov 03 10:28:30.391573 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:30.391541 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7c1aa6_a7c5_4ed6_a871_43a9bf568693.slice/crio-6b6ccf6ff952fbc103bcd1e6f526caecc7523ce358ac57838d3ab7995167f13b WatchSource:0}: Error finding container 6b6ccf6ff952fbc103bcd1e6f526caecc7523ce358ac57838d3ab7995167f13b: Status 404 returned error can't find the container with id 6b6ccf6ff952fbc103bcd1e6f526caecc7523ce358ac57838d3ab7995167f13b Nov 03 10:28:30.411850 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.411824 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk"] Nov 03 10:28:30.423369 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:30.423340 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bf626d_af0f_47d8_8d61_686c390139af.slice/crio-3ca00b168f15cd54b49d1dbea606cfc42ceb5ec7c15efb6b109bae1ff1bcb4b3 WatchSource:0}: Error finding container 3ca00b168f15cd54b49d1dbea606cfc42ceb5ec7c15efb6b109bae1ff1bcb4b3: Status 404 returned error can't find the container with id 3ca00b168f15cd54b49d1dbea606cfc42ceb5ec7c15efb6b109bae1ff1bcb4b3 Nov 03 10:28:30.445204 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.445140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:30.445204 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.445179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:30.445402 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.445281 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 10:28:30.445402 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.445287 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:30.445402 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.445327 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675dcb4c7d-4p5mj: secret "image-registry-tls" not found Nov 03 10:28:30.445402 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.445358 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls podName:7e537c04-9b2a-4d85-9ffd-db08c554c838 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:31.445339748 +0000 UTC m=+31.265963903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69g9h" (UID: "7e537c04-9b2a-4d85-9ffd-db08c554c838") : secret "samples-operator-tls" not found Nov 03 10:28:30.445402 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.445381 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls podName:c634e6ef-e187-4409-a0a0-bae5793bc137 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:31.445364314 +0000 UTC m=+31.265988468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls") pod "image-registry-675dcb4c7d-4p5mj" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137") : secret "image-registry-tls" not found Nov 03 10:28:30.445654 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.445464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:30.445654 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.445512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:30.445654 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.445582 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 10:28:30.445654 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.445596 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:31.445583719 +0000 UTC m=+31.266207873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : configmap references non-existent config key: service-ca.crt Nov 03 10:28:30.445654 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.445619 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:31.445608396 +0000 UTC m=+31.266232535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : secret "router-metrics-certs-default" not found Nov 03 10:28:30.449361 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.449253 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw"] Nov 03 10:28:30.450977 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:30.450945 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86c6b47_2b14_4151_a04a_12c1505e928a.slice/crio-c4b0e1a6cb16382af5ca734b2db6c94add203f2465348d2dbdf1c85142a6166d WatchSource:0}: Error finding container c4b0e1a6cb16382af5ca734b2db6c94add203f2465348d2dbdf1c85142a6166d: Status 404 returned error can't find the container with id c4b0e1a6cb16382af5ca734b2db6c94add203f2465348d2dbdf1c85142a6166d Nov 03 10:28:30.471852 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.471830 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qkwqb"] Nov 03 10:28:30.474538 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:30.474515 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd4ba4e_edba_4732_9f72_54c83b78d6fc.slice/crio-fec64e24bba00eece98e1de08b20b8fed1d37aed0f5bf0467d654ebb87336389 WatchSource:0}: Error finding container fec64e24bba00eece98e1de08b20b8fed1d37aed0f5bf0467d654ebb87336389: Status 404 returned error can't find the container with id fec64e24bba00eece98e1de08b20b8fed1d37aed0f5bf0467d654ebb87336389 Nov 03 10:28:30.510555 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.510524 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf"] Nov 03 10:28:30.514191 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:30.514154 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86d4545_6769_4133_ba67_674e17ab3392.slice/crio-38d40c28f8ac7cf12e57b4eebd78c95fbf271d61abd27cba2cea3e82a2759dbc WatchSource:0}: Error finding container 38d40c28f8ac7cf12e57b4eebd78c95fbf271d61abd27cba2cea3e82a2759dbc: Status 404 returned error can't find the container with id 38d40c28f8ac7cf12e57b4eebd78c95fbf271d61abd27cba2cea3e82a2759dbc Nov 03 10:28:30.546222 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.546197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:30.546614 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.546343 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 10:28:30.546614 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:30.546414 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls podName:f8ea16e9-1545-4881-b7c2-883d8b09a6fe nodeName:}" failed. No retries permitted until 2026-11-03 10:28:31.546396176 +0000 UTC m=+31.367020329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls") pod "dns-default-fb26v" (UID: "f8ea16e9-1545-4881-b7c2-883d8b09a6fe") : secret "dns-default-metrics-tls" not found Nov 03 10:28:30.548412 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.548394 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhr4s"] Nov 03 10:28:30.558226 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:30.558207 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3c3eab_aa4f_4e5b_8b91_1cea0bf245d8.slice/crio-642a45164a990d98917febf0ee367018f01c3675a855436e5f2a675434835b9b WatchSource:0}: Error finding container 642a45164a990d98917febf0ee367018f01c3675a855436e5f2a675434835b9b: Status 404 returned error can't find the container with id 642a45164a990d98917febf0ee367018f01c3675a855436e5f2a675434835b9b Nov 03 10:28:30.909847 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.909769 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhr4s" event={"ID":"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8","Type":"ContainerStarted","Data":"642a45164a990d98917febf0ee367018f01c3675a855436e5f2a675434835b9b"} Nov 03 10:28:30.910963 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.910935 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" event={"ID":"e8bf626d-af0f-47d8-8d61-686c390139af","Type":"ContainerStarted","Data":"3ca00b168f15cd54b49d1dbea606cfc42ceb5ec7c15efb6b109bae1ff1bcb4b3"} Nov 03 10:28:30.912450 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.912410 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wk475" event={"ID":"3e7c1aa6-a7c5-4ed6-a871-43a9bf568693","Type":"ContainerStarted","Data":"4e2708fe88a298838be47de431381d193d0efd89b0947dd31896cb822541eb91"} Nov 03 10:28:30.912547 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.912455 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wk475" event={"ID":"3e7c1aa6-a7c5-4ed6-a871-43a9bf568693","Type":"ContainerStarted","Data":"6b6ccf6ff952fbc103bcd1e6f526caecc7523ce358ac57838d3ab7995167f13b"} Nov 03 10:28:30.914097 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.914074 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" event={"ID":"1dd4ba4e-edba-4732-9f72-54c83b78d6fc","Type":"ContainerStarted","Data":"8de05067f874b69fe96a4e4bcb5480a8070d20b8cb03d5e8e44ae88a903bd6a2"} Nov 03 10:28:30.914182 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.914113 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" event={"ID":"1dd4ba4e-edba-4732-9f72-54c83b78d6fc","Type":"ContainerStarted","Data":"fec64e24bba00eece98e1de08b20b8fed1d37aed0f5bf0467d654ebb87336389"} Nov 03 10:28:30.915478 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.915449 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" event={"ID":"b86c6b47-2b14-4151-a04a-12c1505e928a","Type":"ContainerStarted","Data":"c4b0e1a6cb16382af5ca734b2db6c94add203f2465348d2dbdf1c85142a6166d"} Nov 03 10:28:30.916681 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.916648 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8rp5q" event={"ID":"5f7b33fa-69cf-4582-a4b4-844ed91e2e26","Type":"ContainerStarted","Data":"53b6aeee6f808d6fb6ea455381b1fbb98c8f2fd329357c9c440415b319b9688c"} Nov 03 10:28:30.917891 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.917864 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" event={"ID":"29923b63-cb6e-4a44-b098-ebd0317c1c14","Type":"ContainerStarted","Data":"2da30282695c66f194a8923a7abcdf27b0a6f3ee2028ed120ae5f6bc04141445"} Nov 03 10:28:30.919337 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.919315 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q22k4" event={"ID":"69dd95c4-b504-46d3-97a9-ac864aef3f85","Type":"ContainerStarted","Data":"79eb4cf5d9c1b5c57995a1bb2628d8d648deb6c0b8f3bc0be4cb358ee2a590ab"} Nov 03 10:28:30.919438 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.919352 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q22k4" event={"ID":"69dd95c4-b504-46d3-97a9-ac864aef3f85","Type":"ContainerStarted","Data":"95765fea9b25353b77dc1b85ad2509a2b7987d01d6407e960614d2a1972fb0d3"} Nov 03 10:28:30.920563 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.920534 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" event={"ID":"aedffb88-1606-439b-968d-ab12c5a2b984","Type":"ContainerStarted","Data":"f239d4a549679d7a053791231a64d4299843f672b22739f51676b76b35ec965a"} Nov 03 10:28:30.921639 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.921621 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" event={"ID":"e86d4545-6769-4133-ba67-674e17ab3392","Type":"ContainerStarted","Data":"38d40c28f8ac7cf12e57b4eebd78c95fbf271d61abd27cba2cea3e82a2759dbc"} Nov 03 10:28:30.927635 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.927592 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wk475" podStartSLOduration=416.92757842 podStartE2EDuration="6m56.92757842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:30.92664219 +0000 UTC m=+30.747266356" watchObservedRunningTime="2026-11-03 10:28:30.92757842 +0000 UTC m=+30.748202583" Nov 03 10:28:30.943479 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.943272 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-q22k4" podStartSLOduration=416.943261477 podStartE2EDuration="6m56.943261477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:30.943086428 +0000 UTC m=+30.763710586" watchObservedRunningTime="2026-11-03 10:28:30.943261477 +0000 UTC m=+30.763885638" Nov 03 10:28:30.958316 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:30.958162 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" podStartSLOduration=416.958145616 podStartE2EDuration="6m56.958145616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:30.958110079 +0000 UTC m=+30.778734241" watchObservedRunningTime="2026-11-03 10:28:30.958145616 +0000 UTC m=+30.778769816" Nov 03 10:28:31.355332 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:31.355225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:31.355489 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:31.355336 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:31.355489 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:31.355376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:31.355489 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.355424 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 10:28:31.355630 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.355501 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert podName:cb00dfc2-12f0-4fc3-ba37-0bc83b344602 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:33.355479664 +0000 UTC m=+33.176103822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h65vg" (UID: "cb00dfc2-12f0-4fc3-ba37-0bc83b344602") : secret "networking-console-plugin-cert" not found Nov 03 10:28:31.355630 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.355508 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:31.355630 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.355525 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:31.355630 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.355531 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678cf4bb67-v4rz9: secret "image-registry-tls" not found Nov 03 10:28:31.355630 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.355577 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls podName:5a9404d9-1240-451f-8064-255583bb5da6 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:33.35556208 +0000 UTC m=+33.176186224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rchbs" (UID: "5a9404d9-1240-451f-8064-255583bb5da6") : secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:31.355630 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.355595 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls podName:369e7292-b645-4eb6-9a5d-62e08910f1ef nodeName:}" failed. No retries permitted until 2026-11-03 10:28:33.355585477 +0000 UTC m=+33.176209624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls") pod "image-registry-678cf4bb67-v4rz9" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef") : secret "image-registry-tls" not found Nov 03 10:28:31.456905 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:31.456842 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:31.457081 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:31.456943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:31.457127 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:31.457085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:31.457164 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:31.457117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:31.457272 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.457258 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 10:28:31.457362 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.457342 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls podName:7e537c04-9b2a-4d85-9ffd-db08c554c838 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:33.457321468 +0000 UTC m=+33.277945611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69g9h" (UID: "7e537c04-9b2a-4d85-9ffd-db08c554c838") : secret "samples-operator-tls" not found Nov 03 10:28:31.458075 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.457763 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:33.457747684 +0000 UTC m=+33.278371829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : configmap references non-existent config key: service-ca.crt Nov 03 10:28:31.458075 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.457837 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 10:28:31.458075 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.457903 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:33.457890459 +0000 UTC m=+33.278514605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : secret "router-metrics-certs-default" not found Nov 03 10:28:31.458075 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.457967 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:31.458075 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.458006 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675dcb4c7d-4p5mj: secret "image-registry-tls" not found Nov 03 10:28:31.458075 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.458041 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls podName:c634e6ef-e187-4409-a0a0-bae5793bc137 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:33.458030523 +0000 UTC m=+33.278654723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls") pod "image-registry-675dcb4c7d-4p5mj" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137") : secret "image-registry-tls" not found Nov 03 10:28:31.558223 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:31.558181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:31.558973 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.558453 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 10:28:31.558973 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:31.558519 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls podName:f8ea16e9-1545-4881-b7c2-883d8b09a6fe nodeName:}" failed. No retries permitted until 2026-11-03 10:28:33.558500659 +0000 UTC m=+33.379124813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls") pod "dns-default-fb26v" (UID: "f8ea16e9-1545-4881-b7c2-883d8b09a6fe") : secret "dns-default-metrics-tls" not found Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.378735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.378820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.378854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.379085 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.379172 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls podName:5a9404d9-1240-451f-8064-255583bb5da6 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:37.379146702 +0000 UTC m=+37.199770846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rchbs" (UID: "5a9404d9-1240-451f-8064-255583bb5da6") : secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.379680 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.379744 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert podName:cb00dfc2-12f0-4fc3-ba37-0bc83b344602 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:37.379721517 +0000 UTC m=+37.200345656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h65vg" (UID: "cb00dfc2-12f0-4fc3-ba37-0bc83b344602") : secret "networking-console-plugin-cert" not found Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.379815 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.379828 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678cf4bb67-v4rz9: secret "image-registry-tls" not found Nov 03 10:28:33.379901 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.379865 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls podName:369e7292-b645-4eb6-9a5d-62e08910f1ef nodeName:}" failed. No retries permitted until 2026-11-03 10:28:37.379852629 +0000 UTC m=+37.200476771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls") pod "image-registry-678cf4bb67-v4rz9" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef") : secret "image-registry-tls" not found Nov 03 10:28:33.480331 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.480282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:33.480529 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.480352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:33.480529 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.480434 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:33.480529 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.480438 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:33.480529 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.480452 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675dcb4c7d-4p5mj: secret "image-registry-tls" not found Nov 03 10:28:33.480529 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.480493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:33.480529 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.480514 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls podName:c634e6ef-e187-4409-a0a0-bae5793bc137 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:37.480495363 +0000 UTC m=+37.301119523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls") pod "image-registry-675dcb4c7d-4p5mj" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137") : secret "image-registry-tls" not found Nov 03 10:28:33.480841 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.480551 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 10:28:33.480841 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.480567 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 10:28:33.480841 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.480624 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls podName:7e537c04-9b2a-4d85-9ffd-db08c554c838 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:37.480604894 +0000 UTC m=+37.301229058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69g9h" (UID: "7e537c04-9b2a-4d85-9ffd-db08c554c838") : secret "samples-operator-tls" not found Nov 03 10:28:33.480841 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.480647 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:37.480636708 +0000 UTC m=+37.301260849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : configmap references non-existent config key: service-ca.crt Nov 03 10:28:33.480841 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.480662 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:37.480654478 +0000 UTC m=+37.301278617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : secret "router-metrics-certs-default" not found Nov 03 10:28:33.582561 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.581783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:33.582561 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.582113 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 10:28:33.582561 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:33.582178 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls podName:f8ea16e9-1545-4881-b7c2-883d8b09a6fe nodeName:}" failed. No retries permitted until 2026-11-03 10:28:37.582160613 +0000 UTC m=+37.402784758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls") pod "dns-default-fb26v" (UID: "f8ea16e9-1545-4881-b7c2-883d8b09a6fe") : secret "dns-default-metrics-tls" not found Nov 03 10:28:33.942694 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.942654 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" event={"ID":"b86c6b47-2b14-4151-a04a-12c1505e928a","Type":"ContainerStarted","Data":"afe508fe4619710804a5356bb79771e0af983e64d0904baa68dadcb04e8eaea7"} Nov 03 10:28:33.961471 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:33.960629 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" podStartSLOduration=382.365706093 podStartE2EDuration="6m24.960608431s" totalImagesPullingTime="2.594902338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:33.959562078 +0000 UTC m=+33.780186236" watchObservedRunningTime="2026-11-03 10:28:33.960608431 +0000 UTC m=+33.781232593" Nov 03 10:28:34.391492 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.391405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:34.391956 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:34.391684 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 10:28:34.391956 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:34.391752 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs podName:dcb92e08-27a4-458a-818b-0215acd6dbd5 nodeName:}" failed. No retries permitted until 2026-11-03 10:29:06.391733417 +0000 UTC m=+66.212357559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs") pod "network-metrics-daemon-ztmr2" (UID: "dcb92e08-27a4-458a-818b-0215acd6dbd5") : secret "metrics-daemon-secret" not found Nov 03 10:28:34.492047 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.492012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:34.492204 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.492090 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:34.494910 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.494883 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:34.495802 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.495781 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:34.542821 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.542794 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:34.559531 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.559488 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:34.856095 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.856052 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8sqq"] Nov 03 10:28:34.904871 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.904836 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4vttz"] Nov 03 10:28:34.937396 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:34.937361 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded900688_4264_4ba8_8daa_83e717a6f13f.slice/crio-95b9d82a4dac6860777ab8889652042c7ac53c87baaa71d1c0e696d1aea4105d WatchSource:0}: Error finding container 95b9d82a4dac6860777ab8889652042c7ac53c87baaa71d1c0e696d1aea4105d: Status 404 returned error can't find the container with id 95b9d82a4dac6860777ab8889652042c7ac53c87baaa71d1c0e696d1aea4105d Nov 03 10:28:34.957955 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.957926 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8rp5q" event={"ID":"5f7b33fa-69cf-4582-a4b4-844ed91e2e26","Type":"ContainerStarted","Data":"f8e3943ecc06063845be42fd2416be9fc173fd1c21ff4e900fd4befc89b64f6e"} Nov 03 10:28:34.959330 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:34.959235 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8sqq" event={"ID":"40609c09-1682-4c49-8eae-af9760072fc5","Type":"ContainerStarted","Data":"f40874cd434631d0c82ea59ab46ad95fd6e64bc4acc4c886497ca4e636478f2b"} Nov 03 10:28:35.158757 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.158705 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-8rp5q" podStartSLOduration=382.025795503 podStartE2EDuration="6m26.158688102s" totalImagesPullingTime="4.132892599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:34.976440121 +0000 UTC m=+34.797064285" watchObservedRunningTime="2026-11-03 10:28:35.158688102 +0000 UTC m=+34.979312270" Nov 03 10:28:35.159839 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.159815 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rhvqw"] Nov 03 10:28:35.174404 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.174372 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-2j5wq-fjfdd"] Nov 03 10:28:35.174536 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.174481 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.182133 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.181579 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 10:28:35.182133 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.181649 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 10:28:35.182133 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.181784 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5cc62\"" Nov 03 10:28:35.186770 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.186711 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rhvqw"] Nov 03 10:28:35.186770 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.186765 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2j5wq-fjfdd"] Nov 03 10:28:35.186922 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.186881 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.302479 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.302431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lz4w\" (UniqueName: \"kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.302653 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.302630 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0945eba8-6898-4dfe-ae35-687496c3b596-data-volume\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.302717 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.302685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.302790 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.302767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.302850 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.302814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.302932 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.302905 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.303029 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.302997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n2jd\" (UniqueName: \"kubernetes.io/projected/0945eba8-6898-4dfe-ae35-687496c3b596-kube-api-access-5n2jd\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.303150 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.303079 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.303150 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.303128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0945eba8-6898-4dfe-ae35-687496c3b596-crio-socket\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.404363 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0945eba8-6898-4dfe-ae35-687496c3b596-data-volume\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.404363 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:35.404499 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:35.404568 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:35.904547899 +0000 UTC m=+35.725172039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : secret "insights-runtime-extractor-tls" not found Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:35.404590 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n2jd\" (UniqueName: \"kubernetes.io/projected/0945eba8-6898-4dfe-ae35-687496c3b596-kube-api-access-5n2jd\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:35.404644 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:35.904628 +0000 UTC m=+35.725252152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : configmap "kube-rbac-proxy" not found Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0945eba8-6898-4dfe-ae35-687496c3b596-crio-socket\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lz4w\" (UniqueName: \"kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.404903 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404892 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0945eba8-6898-4dfe-ae35-687496c3b596-data-volume\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.405463 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.404935 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0945eba8-6898-4dfe-ae35-687496c3b596-crio-socket\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.405463 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.405088 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.405463 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.405148 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.405463 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.405384 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.415957 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.415885 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n2jd\" (UniqueName: \"kubernetes.io/projected/0945eba8-6898-4dfe-ae35-687496c3b596-kube-api-access-5n2jd\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.417356 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.417328 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lz4w\" (UniqueName: \"kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.537005 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.536929 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:35.916100 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.915718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.916100 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.915777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:35.916100 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:35.915857 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 10:28:35.916100 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:35.915921 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:36.915904198 +0000 UTC m=+36.736528361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : secret "insights-runtime-extractor-tls" not found Nov 03 10:28:35.916100 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:35.915956 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 10:28:35.916100 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:35.915990 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:36.915980411 +0000 UTC m=+36.736604566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : configmap "kube-rbac-proxy" not found Nov 03 10:28:35.955092 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.954842 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2j5wq-fjfdd"] Nov 03 10:28:35.964540 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:35.964502 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbacc63_2772_4fa1_a868_afc6bc41fa44.slice/crio-c4a2af27c2a8965d975bb507d67b8d7913a9892064d4c08b198143ada77b09ca WatchSource:0}: Error finding container c4a2af27c2a8965d975bb507d67b8d7913a9892064d4c08b198143ada77b09ca: Status 404 returned error can't find the container with id c4a2af27c2a8965d975bb507d67b8d7913a9892064d4c08b198143ada77b09ca Nov 03 10:28:35.975654 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.975039 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" event={"ID":"29923b63-cb6e-4a44-b098-ebd0317c1c14","Type":"ContainerStarted","Data":"95dccbf46d1b84b722664be252773094df825cfd4c39adbc3273686bea37f2d9"} Nov 03 10:28:35.979513 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.979486 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/0.log" Nov 03 10:28:35.979624 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.979525 2565 generic.go:356] "Generic (PLEG): container finished" podID="aedffb88-1606-439b-968d-ab12c5a2b984" containerID="ddfa497ce4e5160896e6d90d61539dcbe91414c36de09bf47bd8eb66181ea65d" exitCode=255 Nov 03 10:28:35.979624 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.979606 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" event={"ID":"aedffb88-1606-439b-968d-ab12c5a2b984","Type":"ContainerDied","Data":"ddfa497ce4e5160896e6d90d61539dcbe91414c36de09bf47bd8eb66181ea65d"} Nov 03 10:28:35.980262 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.979908 2565 scope.go:122] "RemoveContainer" containerID="ddfa497ce4e5160896e6d90d61539dcbe91414c36de09bf47bd8eb66181ea65d" Nov 03 10:28:35.984895 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.984438 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhr4s" event={"ID":"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8","Type":"ContainerStarted","Data":"f29f12ee69fb7ef1bf07581a0205b451afd7eab127f435630e4d104a92948fce"} Nov 03 10:28:35.989734 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.989377 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" event={"ID":"e8bf626d-af0f-47d8-8d61-686c390139af","Type":"ContainerStarted","Data":"bcaa8fc22e86b76bd9ae13f53da0bb7496202c5408d18746a33f84390d8f49da"} Nov 03 10:28:35.992902 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.992791 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" podStartSLOduration=381.667852618 podStartE2EDuration="6m26.992774476s" totalImagesPullingTime="5.324921858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:35.991178865 +0000 UTC m=+35.811803025" watchObservedRunningTime="2026-11-03 10:28:35.992774476 +0000 UTC m=+35.813398638" Nov 03 10:28:35.994023 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:35.993282 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4vttz" event={"ID":"ed900688-4264-4ba8-8daa-83e717a6f13f","Type":"ContainerStarted","Data":"95b9d82a4dac6860777ab8889652042c7ac53c87baaa71d1c0e696d1aea4105d"} Nov 03 10:28:36.036783 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.036669 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jhr4s" podStartSLOduration=1.908024273 podStartE2EDuration="7.036650543s" totalImagesPullingTime="5.12862627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:36.034644469 +0000 UTC m=+35.855268635" watchObservedRunningTime="2026-11-03 10:28:36.036650543 +0000 UTC m=+35.857274699" Nov 03 10:28:36.056333 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.054850 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" podStartSLOduration=381.804702326 podStartE2EDuration="6m27.054831147s" totalImagesPullingTime="5.250128821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:36.054496623 +0000 UTC m=+35.875120785" watchObservedRunningTime="2026-11-03 10:28:36.054831147 +0000 UTC m=+35.875455309" Nov 03 10:28:36.546994 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.545627 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6"] Nov 03 10:28:36.575240 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.574420 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" Nov 03 10:28:36.577222 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.575669 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb"] Nov 03 10:28:36.581013 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.580927 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6fzr5\"" Nov 03 10:28:36.581172 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.581130 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 10:28:36.583135 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.583092 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:36.594940 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.594910 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6"] Nov 03 10:28:36.595054 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.594947 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb"] Nov 03 10:28:36.595054 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.595044 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" Nov 03 10:28:36.665331 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.664934 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4wzkm"] Nov 03 10:28:36.689755 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.689724 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.695287 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.695259 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4wzkm"] Nov 03 10:28:36.695737 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.695715 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 10:28:36.695992 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.695957 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 10:28:36.696129 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.696026 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 10:28:36.696129 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.696041 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 10:28:36.696350 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.696330 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qqtl6\"" Nov 03 10:28:36.725677 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.725643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5v2x\" (UniqueName: \"kubernetes.io/projected/ba906d22-0572-4a25-b444-09b5c553825d-kube-api-access-m5v2x\") pod \"migrator-66df5d496b-xfbpb\" (UID: \"ba906d22-0572-4a25-b444-09b5c553825d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" Nov 03 10:28:36.725836 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.725782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2sdh\" (UniqueName: \"kubernetes.io/projected/ce877b3e-cadd-4cf5-afe7-620e5251c1e3-kube-api-access-z2sdh\") pod \"migrator-66df5d496b-lf8q6\" (UID: \"ce877b3e-cadd-4cf5-afe7-620e5251c1e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" Nov 03 10:28:36.826987 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.826900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2sdh\" (UniqueName: \"kubernetes.io/projected/ce877b3e-cadd-4cf5-afe7-620e5251c1e3-kube-api-access-z2sdh\") pod \"migrator-66df5d496b-lf8q6\" (UID: \"ce877b3e-cadd-4cf5-afe7-620e5251c1e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" Nov 03 10:28:36.827142 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.827006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2zb\" (UniqueName: \"kubernetes.io/projected/03b74c6c-f107-4795-bfbf-a7437c10918e-kube-api-access-gz2zb\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.827142 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.827041 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-cabundle\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.827258 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.827174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5v2x\" (UniqueName: \"kubernetes.io/projected/ba906d22-0572-4a25-b444-09b5c553825d-kube-api-access-m5v2x\") pod \"migrator-66df5d496b-xfbpb\" (UID: \"ba906d22-0572-4a25-b444-09b5c553825d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" Nov 03 10:28:36.827258 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.827203 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-key\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.835261 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.835236 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2sdh\" (UniqueName: \"kubernetes.io/projected/ce877b3e-cadd-4cf5-afe7-620e5251c1e3-kube-api-access-z2sdh\") pod \"migrator-66df5d496b-lf8q6\" (UID: \"ce877b3e-cadd-4cf5-afe7-620e5251c1e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" Nov 03 10:28:36.835416 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.835329 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5v2x\" (UniqueName: \"kubernetes.io/projected/ba906d22-0572-4a25-b444-09b5c553825d-kube-api-access-m5v2x\") pod \"migrator-66df5d496b-xfbpb\" (UID: \"ba906d22-0572-4a25-b444-09b5c553825d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" Nov 03 10:28:36.909748 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.909711 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" Nov 03 10:28:36.928378 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.927847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-key\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.928378 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.928075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:36.928378 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.928128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:36.928378 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.928196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2zb\" (UniqueName: \"kubernetes.io/projected/03b74c6c-f107-4795-bfbf-a7437c10918e-kube-api-access-gz2zb\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.928378 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:36.928241 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 10:28:36.928378 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.928241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-cabundle\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.928378 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:36.928347 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:38.928326919 +0000 UTC m=+38.748951071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : secret "insights-runtime-extractor-tls" not found Nov 03 10:28:36.928378 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:36.928367 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 10:28:36.928826 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:36.928525 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:38.928507675 +0000 UTC m=+38.749131820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : configmap "kube-rbac-proxy" not found Nov 03 10:28:36.928935 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.928915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-cabundle\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.931346 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.931316 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-key\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.933204 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.933166 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" Nov 03 10:28:36.937518 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.937495 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2zb\" (UniqueName: \"kubernetes.io/projected/03b74c6c-f107-4795-bfbf-a7437c10918e-kube-api-access-gz2zb\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:36.998886 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.998848 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" event={"ID":"0bbacc63-2772-4fa1-a868-afc6bc41fa44","Type":"ContainerStarted","Data":"6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba"} Nov 03 10:28:36.999063 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:36.998974 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" event={"ID":"0bbacc63-2772-4fa1-a868-afc6bc41fa44","Type":"ContainerStarted","Data":"c4a2af27c2a8965d975bb507d67b8d7913a9892064d4c08b198143ada77b09ca"} Nov 03 10:28:37.001441 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.001415 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:28:37.002285 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.002219 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/0.log" Nov 03 10:28:37.002419 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.002318 2565 generic.go:356] "Generic (PLEG): container finished" podID="aedffb88-1606-439b-968d-ab12c5a2b984" containerID="47d119010ab7ed8dc33e90f86b5fa1777f750bfedc319ffb5c7860275ab6f22c" exitCode=255 Nov 03 10:28:37.002419 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.002404 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" event={"ID":"aedffb88-1606-439b-968d-ab12c5a2b984","Type":"ContainerDied","Data":"47d119010ab7ed8dc33e90f86b5fa1777f750bfedc319ffb5c7860275ab6f22c"} Nov 03 10:28:37.002531 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.002472 2565 scope.go:122] "RemoveContainer" containerID="ddfa497ce4e5160896e6d90d61539dcbe91414c36de09bf47bd8eb66181ea65d" Nov 03 10:28:37.002685 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.002662 2565 scope.go:122] "RemoveContainer" containerID="47d119010ab7ed8dc33e90f86b5fa1777f750bfedc319ffb5c7860275ab6f22c" Nov 03 10:28:37.002922 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.002901 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m2xr_openshift-console-operator(aedffb88-1606-439b-968d-ab12c5a2b984)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" podUID="aedffb88-1606-439b-968d-ab12c5a2b984" Nov 03 10:28:37.014587 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.014534 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" podStartSLOduration=2.014521327 podStartE2EDuration="2.014521327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:37.013938728 +0000 UTC m=+36.834562889" watchObservedRunningTime="2026-11-03 10:28:37.014521327 +0000 UTC m=+36.835145488" Nov 03 10:28:37.015979 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.015957 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:37.342081 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.342043 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb"] Nov 03 10:28:37.346909 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:37.346878 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba906d22_0572_4a25_b444_09b5c553825d.slice/crio-62620f06b1463de681bce804b51273054ea05e1c04fbf9b8bcd0b40d52b2b8e6 WatchSource:0}: Error finding container 62620f06b1463de681bce804b51273054ea05e1c04fbf9b8bcd0b40d52b2b8e6: Status 404 returned error can't find the container with id 62620f06b1463de681bce804b51273054ea05e1c04fbf9b8bcd0b40d52b2b8e6 Nov 03 10:28:37.373401 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.373321 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6"] Nov 03 10:28:37.376236 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.376207 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4wzkm"] Nov 03 10:28:37.386140 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:37.386026 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce877b3e_cadd_4cf5_afe7_620e5251c1e3.slice/crio-14c5e03cffb3143d08863d35eef13abdaecc9ea44b96338a4c7dcfc99e4c0244 WatchSource:0}: Error finding container 14c5e03cffb3143d08863d35eef13abdaecc9ea44b96338a4c7dcfc99e4c0244: Status 404 returned error can't find the container with id 14c5e03cffb3143d08863d35eef13abdaecc9ea44b96338a4c7dcfc99e4c0244 Nov 03 10:28:37.398751 ip-10-0-139-90 kubenswrapper[2565]: W1103 10:28:37.388780 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b74c6c_f107_4795_bfbf_a7437c10918e.slice/crio-b4b61720243e76813094ed0013ab238f52b29e0161b2915b295dde84ee491d73 WatchSource:0}: Error finding container b4b61720243e76813094ed0013ab238f52b29e0161b2915b295dde84ee491d73: Status 404 returned error can't find the container with id b4b61720243e76813094ed0013ab238f52b29e0161b2915b295dde84ee491d73 Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.434476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.434539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.434579 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.434747 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.434809 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls podName:5a9404d9-1240-451f-8064-255583bb5da6 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:45.434791486 +0000 UTC m=+45.255415630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rchbs" (UID: "5a9404d9-1240-451f-8064-255583bb5da6") : secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.434912 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.434947 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert podName:cb00dfc2-12f0-4fc3-ba37-0bc83b344602 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:45.4349362 +0000 UTC m=+45.255560343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-h65vg" (UID: "cb00dfc2-12f0-4fc3-ba37-0bc83b344602") : secret "networking-console-plugin-cert" not found Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.435005 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.435016 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678cf4bb67-v4rz9: secret "image-registry-tls" not found Nov 03 10:28:37.435375 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.435046 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls podName:369e7292-b645-4eb6-9a5d-62e08910f1ef nodeName:}" failed. No retries permitted until 2026-11-03 10:28:45.435035625 +0000 UTC m=+45.255659768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls") pod "image-registry-678cf4bb67-v4rz9" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef") : secret "image-registry-tls" not found Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.535992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.536044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.536125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.536148 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.536166 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675dcb4c7d-4p5mj: secret "image-registry-tls" not found Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.536180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.536225 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls podName:c634e6ef-e187-4409-a0a0-bae5793bc137 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:45.536207044 +0000 UTC m=+45.356831208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls") pod "image-registry-675dcb4c7d-4p5mj" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137") : secret "image-registry-tls" not found Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.536249 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.536287 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.536330 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls podName:7e537c04-9b2a-4d85-9ffd-db08c554c838 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:45.536311663 +0000 UTC m=+45.356935819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69g9h" (UID: "7e537c04-9b2a-4d85-9ffd-db08c554c838") : secret "samples-operator-tls" not found Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.536352 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:45.536341425 +0000 UTC m=+45.356965590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : secret "router-metrics-certs-default" not found Nov 03 10:28:37.536419 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.536367 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:45.53635997 +0000 UTC m=+45.356984114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : configmap references non-existent config key: service-ca.crt Nov 03 10:28:37.637256 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:37.637164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:37.637729 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.637463 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 10:28:37.637729 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:37.637531 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls podName:f8ea16e9-1545-4881-b7c2-883d8b09a6fe nodeName:}" failed. No retries permitted until 2026-11-03 10:28:45.637512325 +0000 UTC m=+45.458136487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls") pod "dns-default-fb26v" (UID: "f8ea16e9-1545-4881-b7c2-883d8b09a6fe") : secret "dns-default-metrics-tls" not found Nov 03 10:28:38.008576 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.008498 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:28:38.009623 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.009599 2565 scope.go:122] "RemoveContainer" containerID="47d119010ab7ed8dc33e90f86b5fa1777f750bfedc319ffb5c7860275ab6f22c" Nov 03 10:28:38.009880 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:38.009846 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m2xr_openshift-console-operator(aedffb88-1606-439b-968d-ab12c5a2b984)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" podUID="aedffb88-1606-439b-968d-ab12c5a2b984" Nov 03 10:28:38.011912 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.011885 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" event={"ID":"e86d4545-6769-4133-ba67-674e17ab3392","Type":"ContainerStarted","Data":"b7b52dd826671dc278d9977907ff13aa98fc54c3a196da738f683f0ae140bd8f"} Nov 03 10:28:38.015491 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.015469 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" event={"ID":"ba906d22-0572-4a25-b444-09b5c553825d","Type":"ContainerStarted","Data":"62620f06b1463de681bce804b51273054ea05e1c04fbf9b8bcd0b40d52b2b8e6"} Nov 03 10:28:38.019320 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.019131 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" event={"ID":"03b74c6c-f107-4795-bfbf-a7437c10918e","Type":"ContainerStarted","Data":"9cbfd5392d89cbe04046cfe84d11bca55dc17597b6b82f4107cce90dffa17fd7"} Nov 03 10:28:38.019418 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.019253 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" event={"ID":"03b74c6c-f107-4795-bfbf-a7437c10918e","Type":"ContainerStarted","Data":"b4b61720243e76813094ed0013ab238f52b29e0161b2915b295dde84ee491d73"} Nov 03 10:28:38.021008 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.020981 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" event={"ID":"ce877b3e-cadd-4cf5-afe7-620e5251c1e3","Type":"ContainerStarted","Data":"14c5e03cffb3143d08863d35eef13abdaecc9ea44b96338a4c7dcfc99e4c0244"} Nov 03 10:28:38.053610 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.053552 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" podStartSLOduration=2.053533481 podStartE2EDuration="2.053533481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:38.052750296 +0000 UTC m=+37.873374458" watchObservedRunningTime="2026-11-03 10:28:38.053533481 +0000 UTC m=+37.874157644" Nov 03 10:28:38.071399 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.070983 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" podStartSLOduration=382.518838382 podStartE2EDuration="6m29.07096614s" totalImagesPullingTime="6.552127758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:38.069530709 +0000 UTC m=+37.890154871" watchObservedRunningTime="2026-11-03 10:28:38.07096614 +0000 UTC m=+37.891590302" Nov 03 10:28:38.968327 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.967478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:38.968327 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:38.967535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:38.968327 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:38.967633 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 10:28:38.968327 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:38.967693 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:42.967675259 +0000 UTC m=+42.788299405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : configmap "kube-rbac-proxy" not found Nov 03 10:28:38.968327 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:38.968114 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 10:28:38.968327 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:38.968159 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:42.968146199 +0000 UTC m=+42.788770344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : secret "insights-runtime-extractor-tls" not found Nov 03 10:28:39.039699 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:39.039606 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8sqq" event={"ID":"40609c09-1682-4c49-8eae-af9760072fc5","Type":"ContainerStarted","Data":"23ea3c90852545df332b57d3cbe0922260b348f5465ab205748416ec024b0218"} Nov 03 10:28:39.058198 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:39.058055 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t8sqq" podStartSLOduration=34.292751126 podStartE2EDuration="38.058040303s" totalImagesPullingTime="3.765289177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:39.056555846 +0000 UTC m=+38.877180007" watchObservedRunningTime="2026-11-03 10:28:39.058040303 +0000 UTC m=+38.878664464" Nov 03 10:28:39.444391 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:39.444369 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jhp98_35d6f6db-753a-4d1e-ae56-58659a5baab1/dns-node-resolver/0.log" Nov 03 10:28:39.970805 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:39.970768 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:39.971210 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:39.970844 2565 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:39.971278 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:39.971255 2565 scope.go:122] "RemoveContainer" containerID="47d119010ab7ed8dc33e90f86b5fa1777f750bfedc319ffb5c7860275ab6f22c" Nov 03 10:28:39.971505 ip-10-0-139-90 kubenswrapper[2565]: E1103 10:28:39.971483 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2m2xr_openshift-console-operator(aedffb88-1606-439b-968d-ab12c5a2b984)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" podUID="aedffb88-1606-439b-968d-ab12c5a2b984" Nov 03 10:28:40.045753 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:40.045676 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" event={"ID":"ba906d22-0572-4a25-b444-09b5c553825d","Type":"ContainerStarted","Data":"8b411fb4cf862389abc22cc3514e452725e125fabbf94e8f4e058248e862cdbd"} Nov 03 10:28:40.048042 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:40.048005 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" event={"ID":"ce877b3e-cadd-4cf5-afe7-620e5251c1e3","Type":"ContainerStarted","Data":"8e773066a976833c3d634a0e68304dfc337c3f08bb6d09d989d08c561f356e89"} Nov 03 10:28:40.048189 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:40.048148 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:40.061471 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:40.061413 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" podStartSLOduration=2.024904931 podStartE2EDuration="4.061396093s" totalImagesPullingTime="2.036491162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:40.060598337 +0000 UTC m=+39.881222500" watchObservedRunningTime="2026-11-03 10:28:40.061396093 +0000 UTC m=+39.882020254" Nov 03 10:28:40.105017 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:40.104962 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" podStartSLOduration=2.110776746 podStartE2EDuration="4.10494752s" totalImagesPullingTime="1.994170774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:40.104071295 +0000 UTC m=+39.924695458" watchObservedRunningTime="2026-11-03 10:28:40.10494752 +0000 UTC m=+39.925571681" Nov 03 10:28:40.444720 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:40.444643 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ssnbk_35dabb24-0583-4de3-9a5d-21c8ef04c7c5/node-ca/0.log" Nov 03 10:28:40.846599 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:40.846511 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhr4s_4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8/serve-healthcheck-canary/0.log" Nov 03 10:28:41.045568 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:41.045537 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lf8q6_ce877b3e-cadd-4cf5-afe7-620e5251c1e3/migrator/0.log" Nov 03 10:28:41.245695 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:41.245617 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-xfbpb_ba906d22-0572-4a25-b444-09b5c553825d/migrator/0.log" Nov 03 10:28:41.446685 ip-10-0-139-90 kubenswrapper[2565]: I1103 10:28:41.446657 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f_29923b63-cb6e-4a44-b098-ebd0317c1c14/kube-storage-version-migrator-operator/0.log" Nov 03 10:28:41.942290 ip-10-0-139-90 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 10:28:41.955725 ip-10-0-139-90 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 10:28:41.955895 ip-10-0-139-90 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 10:28:41.955944 ip-10-0-139-90 systemd[1]: kubelet.service: Consumed 3.331s CPU time. Nov 03 10:28:41.966777 ip-10-0-139-90 systemd[1]: Starting Kubernetes Kubelet... Nov 03 10:28:42.079226 ip-10-0-139-90 kubenswrapper[5086]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 10:28:42.079226 ip-10-0-139-90 kubenswrapper[5086]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 10:28:42.079226 ip-10-0-139-90 kubenswrapper[5086]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 10:28:42.079226 ip-10-0-139-90 kubenswrapper[5086]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 10:28:42.082535 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082434 5086 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:42.082535 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082532 5086 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:42.082535 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082537 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:42.082535 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082541 5086 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082545 5086 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082548 5086 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082550 5086 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082554 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082557 5086 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082559 5086 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082562 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082565 5086 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082568 5086 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082571 5086 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082573 5086 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082576 5086 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082579 5086 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082582 5086 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082585 5086 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082587 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082590 5086 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082592 5086 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:42.082713 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082595 5086 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082597 5086 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082600 5086 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082603 5086 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082605 5086 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082608 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082611 5086 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082613 5086 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082616 5086 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082620 5086 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082622 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082625 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082628 5086 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082631 5086 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082634 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082637 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082640 5086 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082643 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082645 5086 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:42.083172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082648 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082651 5086 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082653 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082656 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082659 5086 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082662 5086 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082665 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082668 5086 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082671 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082673 5086 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082676 5086 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082679 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082681 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082684 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082686 5086 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082689 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082692 5086 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082694 5086 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082697 5086 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:42.083654 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082705 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082708 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082711 5086 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082716 5086 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082721 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082725 5086 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082728 5086 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082731 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082734 5086 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082736 5086 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082739 5086 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082742 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082745 5086 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082747 5086 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082750 5086 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082753 5086 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082757 5086 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082759 5086 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082762 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:42.084119 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082764 5086 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082767 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082770 5086 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082773 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082777 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082779 5086 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082782 5086 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082785 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082788 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082790 5086 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082793 5086 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082795 5086 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082799 5086 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082802 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082804 5086 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082807 5086 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082810 5086 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082813 5086 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082816 5086 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082819 5086 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:42.084607 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082822 5086 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082824 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082827 5086 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082830 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082835 5086 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082839 5086 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.082842 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083672 5086 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083681 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083684 5086 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083687 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083690 5086 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083693 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083696 5086 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083699 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083703 5086 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083707 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083711 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083714 5086 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:42.085107 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083717 5086 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083720 5086 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083723 5086 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083725 5086 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083728 5086 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083731 5086 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083735 5086 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083737 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083740 5086 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083743 5086 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083749 5086 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083752 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083755 5086 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083758 5086 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083760 5086 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083763 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083766 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083768 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083771 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083774 5086 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:42.085583 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083776 5086 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083779 5086 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083782 5086 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083785 5086 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083787 5086 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083790 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083793 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083795 5086 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083798 5086 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083801 5086 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083803 5086 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083806 5086 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083809 5086 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083812 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083815 5086 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083817 5086 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083820 5086 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083824 5086 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083827 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083829 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:42.086098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083832 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083834 5086 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083838 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083842 5086 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083846 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083849 5086 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083852 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083855 5086 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083857 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083860 5086 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083863 5086 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083865 5086 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083868 5086 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083871 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083873 5086 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083876 5086 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083879 5086 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083881 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083884 5086 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:42.086604 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083887 5086 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083889 5086 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083892 5086 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083894 5086 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083897 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083901 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083903 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083907 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083910 5086 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083913 5086 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083916 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083918 5086 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083921 5086 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083924 5086 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083926 5086 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083930 5086 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083933 5086 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083936 5086 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083939 5086 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:42.087069 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083941 5086 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083944 5086 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083947 5086 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083949 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083952 5086 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083955 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083958 5086 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083961 5086 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083964 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083966 5086 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083969 5086 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083972 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083974 5086 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083977 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083980 5086 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.083983 5086 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084510 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084518 5086 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084522 5086 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084525 5086 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:42.087575 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084529 5086 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084532 5086 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084535 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084537 5086 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084540 5086 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084543 5086 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084546 5086 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084549 5086 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084552 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084555 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084557 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084560 5086 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084563 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084565 5086 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084567 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084570 5086 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084573 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084576 5086 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084579 5086 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:42.088086 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084583 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084585 5086 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084588 5086 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084591 5086 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084593 5086 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084596 5086 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084599 5086 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084601 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084604 5086 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084607 5086 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084610 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084613 5086 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084617 5086 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084620 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084622 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084625 5086 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084628 5086 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084630 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084633 5086 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:42.088603 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084636 5086 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084638 5086 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084641 5086 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084643 5086 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084646 5086 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084649 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084651 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084654 5086 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084657 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084659 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084662 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084665 5086 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084668 5086 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084670 5086 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084673 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084676 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084679 5086 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084682 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084684 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084687 5086 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:42.089082 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084689 5086 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084692 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084696 5086 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084699 5086 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084702 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084705 5086 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084709 5086 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084712 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084715 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084718 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084720 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084723 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084725 5086 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084728 5086 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084731 5086 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084734 5086 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084736 5086 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084739 5086 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084741 5086 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:42.089593 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084744 5086 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084746 5086 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084749 5086 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084752 5086 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084755 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084758 5086 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084761 5086 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084764 5086 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084767 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084769 5086 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084772 5086 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084774 5086 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084777 5086 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084780 5086 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084783 5086 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084789 5086 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084791 5086 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084794 5086 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084797 5086 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084800 5086 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:42.090052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084803 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084806 5086 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084808 5086 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084811 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.084814 5086 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084908 5086 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084923 5086 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084930 5086 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084937 5086 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084942 5086 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084945 5086 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084950 5086 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084954 5086 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084958 5086 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084961 5086 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084965 5086 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084968 5086 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084971 5086 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084974 5086 flags.go:64] FLAG: --cgroup-root="" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084978 5086 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084981 5086 flags.go:64] FLAG: --client-ca-file="" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084984 5086 flags.go:64] FLAG: --cloud-provider="external" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084987 5086 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084991 5086 flags.go:64] FLAG: --cluster-domain="" Nov 03 10:28:42.090553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.084995 5086 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085000 5086 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085005 5086 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085014 5086 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085020 5086 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085025 5086 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085030 5086 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085034 5086 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085037 5086 flags.go:64] FLAG: --contention-profiling="false" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085041 5086 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085045 5086 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085048 5086 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085051 5086 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085056 5086 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085059 5086 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085062 5086 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085065 5086 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085068 5086 flags.go:64] FLAG: --enable-server="true" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085071 5086 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085076 5086 flags.go:64] FLAG: --event-burst="100" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085079 5086 flags.go:64] FLAG: --event-qps="50" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085082 5086 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085086 5086 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085089 5086 flags.go:64] FLAG: --eviction-hard="" Nov 03 10:28:42.091114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085093 5086 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085096 5086 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085099 5086 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085103 5086 flags.go:64] FLAG: --eviction-soft="" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085106 5086 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085109 5086 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085112 5086 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085116 5086 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085119 5086 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085122 5086 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085125 5086 flags.go:64] FLAG: --feature-gates="" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085131 5086 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085134 5086 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085138 5086 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085141 5086 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085144 5086 flags.go:64] FLAG: --healthz-port="10248" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085147 5086 flags.go:64] FLAG: --help="false" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085150 5086 flags.go:64] FLAG: --hostname-override="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085155 5086 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085158 5086 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085161 5086 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085164 5086 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085168 5086 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 10:28:42.091723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085171 5086 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085174 5086 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085177 5086 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085181 5086 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085184 5086 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085188 5086 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085191 5086 flags.go:64] FLAG: --kube-reserved="" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085194 5086 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085197 5086 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085200 5086 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085203 5086 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085206 5086 flags.go:64] FLAG: --lock-file="" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085210 5086 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085213 5086 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085216 5086 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085222 5086 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085226 5086 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085229 5086 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085232 5086 flags.go:64] FLAG: --logging-format="text" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085235 5086 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085240 5086 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085243 5086 flags.go:64] FLAG: --manifest-url="" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085246 5086 flags.go:64] FLAG: --manifest-url-header="" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085251 5086 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085254 5086 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 10:28:42.092290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085258 5086 flags.go:64] FLAG: --max-pods="110" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085262 5086 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085265 5086 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085269 5086 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085272 5086 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085275 5086 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085278 5086 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085281 5086 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085289 5086 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085303 5086 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085307 5086 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085310 5086 flags.go:64] FLAG: --pod-cidr="" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085313 5086 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085316 5086 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085319 5086 flags.go:64] FLAG: --pods-per-core="0" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085322 5086 flags.go:64] FLAG: --port="10250" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085325 5086 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085329 5086 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d6fda63dfe66079c" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085332 5086 flags.go:64] FLAG: --qos-reserved="" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085335 5086 flags.go:64] FLAG: --read-only-port="10255" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085348 5086 flags.go:64] FLAG: --register-node="true" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085351 5086 flags.go:64] FLAG: --register-with-taints="" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085356 5086 flags.go:64] FLAG: --registry-burst="10" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085360 5086 flags.go:64] FLAG: --registry-qps="5" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085365 5086 flags.go:64] FLAG: --reserved-cpus="" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085369 5086 flags.go:64] FLAG: --reserved-memory="" Nov 03 10:28:42.092963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085375 5086 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085381 5086 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085384 5086 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085387 5086 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085390 5086 flags.go:64] FLAG: --runonce="false" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085393 5086 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085396 5086 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085399 5086 flags.go:64] FLAG: --seccomp-default="false" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085402 5086 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085405 5086 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085409 5086 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085413 5086 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085416 5086 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085419 5086 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085424 5086 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085427 5086 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085430 5086 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085433 5086 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085436 5086 flags.go:64] FLAG: --system-cgroups="" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085439 5086 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085445 5086 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085448 5086 flags.go:64] FLAG: --tls-cert-file="" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085451 5086 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085455 5086 flags.go:64] FLAG: --tls-min-version="" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085458 5086 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 10:28:42.093725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085461 5086 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085464 5086 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085467 5086 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085470 5086 flags.go:64] FLAG: --v="2" Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085475 5086 flags.go:64] FLAG: --version="false" Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085480 5086 flags.go:64] FLAG: --vmodule="" Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085484 5086 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085487 5086 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085636 5086 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085640 5086 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085643 5086 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085646 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085649 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085654 5086 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085658 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085661 5086 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085665 5086 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085668 5086 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085671 5086 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085674 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:42.094372 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085676 5086 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085680 5086 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085683 5086 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085686 5086 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085688 5086 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085691 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085694 5086 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085697 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085699 5086 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085702 5086 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085705 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085708 5086 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085710 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085713 5086 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085716 5086 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085719 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085721 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085725 5086 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085727 5086 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:42.095008 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085731 5086 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085735 5086 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085738 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085740 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085743 5086 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085746 5086 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085749 5086 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085751 5086 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085754 5086 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085757 5086 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085759 5086 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085762 5086 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085767 5086 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085771 5086 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085774 5086 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085777 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085780 5086 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085782 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085785 5086 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085788 5086 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:42.095548 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085791 5086 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085793 5086 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085796 5086 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085799 5086 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085802 5086 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085804 5086 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085807 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085810 5086 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085813 5086 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085815 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085818 5086 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085821 5086 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085823 5086 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085828 5086 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085830 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085833 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085836 5086 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085839 5086 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085842 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085845 5086 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:42.096133 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085847 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085850 5086 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085853 5086 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085856 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085858 5086 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085867 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085870 5086 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085873 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085875 5086 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085878 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085881 5086 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085883 5086 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085886 5086 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085889 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085891 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085894 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085897 5086 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085900 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085902 5086 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:42.096949 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085905 5086 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085908 5086 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085910 5086 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085913 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085916 5086 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085918 5086 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085922 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085925 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085928 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085930 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085933 5086 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085936 5086 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085939 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085942 5086 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085944 5086 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.085947 5086 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:42.097708 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.085973 5086 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086019 5086 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086024 5086 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086034 5086 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086037 5086 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086041 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086043 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086046 5086 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086049 5086 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086052 5086 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086054 5086 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086057 5086 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086060 5086 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086062 5086 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086065 5086 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086069 5086 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086073 5086 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086076 5086 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086079 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086082 5086 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:42.098211 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086084 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086087 5086 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086090 5086 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086094 5086 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086096 5086 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086100 5086 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086104 5086 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086107 5086 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086110 5086 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086113 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086116 5086 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086118 5086 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086121 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086124 5086 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086126 5086 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086129 5086 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086137 5086 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086140 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086143 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086147 5086 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:42.098759 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086149 5086 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086152 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086155 5086 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086157 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086160 5086 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086163 5086 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086165 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086168 5086 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086170 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086174 5086 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086176 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086180 5086 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086183 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086186 5086 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086188 5086 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086195 5086 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086198 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086201 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086204 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:42.099288 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086207 5086 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086210 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086213 5086 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086215 5086 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086218 5086 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086221 5086 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086224 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086226 5086 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086229 5086 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086232 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086236 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086238 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086241 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086244 5086 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086247 5086 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086249 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086252 5086 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086255 5086 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086258 5086 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086260 5086 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:42.099816 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086263 5086 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086266 5086 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086269 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086271 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086275 5086 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086279 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086282 5086 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086285 5086 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086289 5086 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086304 5086 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086307 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086311 5086 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086314 5086 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086317 5086 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086320 5086 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086323 5086 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086325 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086328 5086 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:42.100378 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086331 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086334 5086 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086337 5086 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086339 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086342 5086 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086345 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086348 5086 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086351 5086 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086354 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.086356 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.090322 5086 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.090338 5086 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.090357 5086 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.090362 5086 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090420 5086 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090425 5086 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090429 5086 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090432 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090436 5086 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:42.100857 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090438 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090441 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090444 5086 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090447 5086 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090450 5086 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090452 5086 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090455 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090458 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090460 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090463 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090466 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090468 5086 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090471 5086 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090474 5086 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090477 5086 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090480 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090483 5086 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090486 5086 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090489 5086 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090492 5086 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:42.101330 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090494 5086 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090497 5086 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090500 5086 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090504 5086 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090516 5086 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090520 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090524 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090528 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090532 5086 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090536 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090540 5086 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090545 5086 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090550 5086 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090556 5086 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090563 5086 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090567 5086 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090571 5086 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090574 5086 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090577 5086 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:42.101823 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090579 5086 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090582 5086 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090585 5086 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090588 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090590 5086 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090593 5086 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090596 5086 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090599 5086 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090603 5086 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090606 5086 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090608 5086 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090611 5086 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090614 5086 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090617 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090619 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090622 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090625 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090629 5086 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090640 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:42.102307 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090644 5086 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090647 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090650 5086 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090653 5086 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090656 5086 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090659 5086 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090662 5086 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090665 5086 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090667 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090670 5086 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090672 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090675 5086 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090678 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090680 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090683 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090686 5086 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090689 5086 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090691 5086 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090694 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090697 5086 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:42.102775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090699 5086 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090702 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090705 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090708 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090711 5086 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090714 5086 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090716 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090719 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090722 5086 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090725 5086 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090727 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090730 5086 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090738 5086 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090741 5086 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090744 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090747 5086 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090749 5086 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090752 5086 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090754 5086 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:42.103265 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090757 5086 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090760 5086 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090762 5086 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090765 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.090794 5086 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090993 5086 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.090997 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091000 5086 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091003 5086 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091006 5086 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091010 5086 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091015 5086 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091019 5086 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091022 5086 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091025 5086 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:42.103731 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091028 5086 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091030 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091034 5086 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091036 5086 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091039 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091042 5086 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091044 5086 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091047 5086 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091050 5086 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091052 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091055 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091064 5086 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091067 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091070 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091073 5086 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091075 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091078 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091080 5086 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091083 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091086 5086 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:42.104098 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091089 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091091 5086 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091094 5086 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091096 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091099 5086 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091102 5086 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091104 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091107 5086 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091110 5086 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091113 5086 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091115 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091118 5086 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091121 5086 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091124 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091128 5086 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091138 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091142 5086 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091145 5086 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091148 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091150 5086 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:42.104674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091153 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091155 5086 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091158 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091161 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091169 5086 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091172 5086 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091175 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091178 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091180 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091183 5086 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091186 5086 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091189 5086 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091192 5086 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091195 5086 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091197 5086 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091200 5086 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091202 5086 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091205 5086 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091208 5086 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:42.105187 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091210 5086 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091213 5086 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091216 5086 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091218 5086 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091221 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091224 5086 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091226 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091229 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091232 5086 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091235 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091237 5086 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091240 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091243 5086 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091245 5086 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091248 5086 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091251 5086 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091253 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091256 5086 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091264 5086 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091267 5086 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:42.105838 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091270 5086 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091273 5086 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091275 5086 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091278 5086 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091281 5086 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091283 5086 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091286 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091289 5086 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091304 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091309 5086 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091312 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091314 5086 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091317 5086 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091320 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091323 5086 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091326 5086 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:42.106345 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091328 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.091354 5086 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091397 5086 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091401 5086 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091405 5086 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091407 5086 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091410 5086 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091412 5086 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091415 5086 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091418 5086 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091420 5086 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091423 5086 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091426 5086 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091429 5086 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091432 5086 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091440 5086 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 10:28:42.106778 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091443 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091445 5086 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091448 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091450 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091453 5086 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091455 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091458 5086 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091461 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091463 5086 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091466 5086 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091469 5086 feature_gate.go:386] unrecognized feature gate: Example Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091471 5086 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091474 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091477 5086 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091480 5086 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091483 5086 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091485 5086 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091488 5086 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091491 5086 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091493 5086 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 10:28:42.107177 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091496 5086 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091499 5086 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091501 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091504 5086 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091507 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091509 5086 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091512 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091514 5086 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091517 5086 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091520 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091522 5086 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091525 5086 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091533 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091536 5086 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091539 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091541 5086 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091544 5086 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091547 5086 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091551 5086 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 10:28:42.107685 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091554 5086 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091560 5086 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091563 5086 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091566 5086 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091569 5086 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091571 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091575 5086 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091578 5086 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091580 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091583 5086 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091586 5086 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091588 5086 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091591 5086 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091593 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091596 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091599 5086 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091601 5086 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091604 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091606 5086 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091613 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 10:28:42.108196 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091615 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091618 5086 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091621 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091623 5086 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091626 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091629 5086 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091637 5086 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091640 5086 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091643 5086 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091646 5086 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091648 5086 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091651 5086 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091654 5086 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091658 5086 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091660 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091663 5086 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091666 5086 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091668 5086 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091672 5086 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 10:28:42.108691 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091676 5086 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091679 5086 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091682 5086 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091684 5086 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091687 5086 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091690 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091693 5086 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091695 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091698 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091700 5086 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091703 5086 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091706 5086 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091710 5086 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:42.091713 5086 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.091780 5086 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.092783 5086 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.092845 5086 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.093048 5086 server.go:1014] "Starting client certificate rotation" Nov 03 10:28:42.109166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.093213 5086 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 10:28:42.109612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.093266 5086 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 10:23:00 +0000 UTC" deadline="2028-08-11 17:02:51.823354498 +0000 UTC" Nov 03 10:28:42.109612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.093347 5086 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15534h34m9.730010781s" Nov 03 10:28:42.109612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.093726 5086 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 10:28:42.109612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.095473 5086 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 10:28:42.109612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.097539 5086 log.go:25] "Validated CRI v1 runtime API" Nov 03 10:28:42.109612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.099376 5086 log.go:25] "Validated CRI v1 image API" Nov 03 10:28:42.109612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.100175 5086 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 10:28:42.109612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.102323 5086 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bdad2ae6-4a9a-40a6-816f-b33c3fbae43f:/dev/nvme0n1p4 d0ba2c1a-02ed-4fa8-911a-41c7405de004:/dev/nvme0n1p3] Nov 03 10:28:42.109854 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.102344 5086 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/051e792f9023c91f8519b920f269eb39e1f6a1199e8fdc0c4d1a142f9f9b26c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/051e792f9023c91f8519b920f269eb39e1f6a1199e8fdc0c4d1a142f9f9b26c2/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e448fcae41e1f65a439d0cefbd87e6bf0d76f7f785a5a3591fafcf7e0e4bbc4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e448fcae41e1f65a439d0cefbd87e6bf0d76f7f785a5a3591fafcf7e0e4bbc4/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14c5e03cffb3143d08863d35eef13abdaecc9ea44b96338a4c7dcfc99e4c0244/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14c5e03cffb3143d08863d35eef13abdaecc9ea44b96338a4c7dcfc99e4c0244/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2da30282695c66f194a8923a7abcdf27b0a6f3ee2028ed120ae5f6bc04141445/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2da30282695c66f194a8923a7abcdf27b0a6f3ee2028ed120ae5f6bc04141445/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38d40c28f8ac7cf12e57b4eebd78c95fbf271d61abd27cba2cea3e82a2759dbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38d40c28f8ac7cf12e57b4eebd78c95fbf271d61abd27cba2cea3e82a2759dbc/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ca00b168f15cd54b49d1dbea606cfc42ceb5ec7c15efb6b109bae1ff1bcb4b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ca00b168f15cd54b49d1dbea606cfc42ceb5ec7c15efb6b109bae1ff1bcb4b3/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53b6aeee6f808d6fb6ea455381b1fbb98c8f2fd329357c9c440415b319b9688c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53b6aeee6f808d6fb6ea455381b1fbb98c8f2fd329357c9c440415b319b9688c/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/596e649f74a93fff5772be92eac7e59f0cd8d95345040fce7338e9e66435da4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/596e649f74a93fff5772be92eac7e59f0cd8d95345040fce7338e9e66435da4c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62620f06b1463de681bce804b51273054ea05e1c04fbf9b8bcd0b40d52b2b8e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62620f06b1463de681bce804b51273054ea05e1c04fbf9b8bcd0b40d52b2b8e6/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/642a45164a990d98917febf0ee367018f01c3675a855436e5f2a675434835b9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/642a45164a990d98917febf0ee367018f01c3675a855436e5f2a675434835b9b/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b6ccf6ff952fbc103bcd1e6f526caecc7523ce358ac57838d3ab7995167f13b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b6ccf6ff952fbc103bcd1e6f526caecc7523ce358ac57838d3ab7995167f13b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71eff809985da4d6fcb78437170620f5a5f97bb544ab28f0ba74970ef67200c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71eff809985da4d6fcb78437170620f5a5f97bb544ab28f0ba74970ef67200c4/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/885a845459902eae25f1863fe55ea16efd70b3aae005cd200792d5535322a573/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/885a845459902eae25f1863fe55ea16efd70b3aae005cd200792d5535322a573/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95765fea9b25353b77dc1b85ad2509a2b7987d01d6407e960614d2a1972fb0d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95765fea9b25353b77dc1b85ad2509a2b7987d01d6407e960614d2a1972fb0d3/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95b9d82a4dac6860777ab8889652042c7ac53c87baaa71d1c0e696d1aea4105d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95b9d82a4dac6860777ab8889652042c7ac53c87baaa71d1c0e696d1aea4105d/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4597f40640b48da624e13186a05078f486220e826d4ed2e16cef82684f15f8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4597f40640b48da624e13186a05078f486220e826d4ed2e16cef82684f15f8a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b35e39ed7b164fda5ca6fb8cd155503010d6e252f0ad4185ff348c7ba33fd8f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b35e39ed7b164fda5ca6fb8cd155503010d6e252f0ad4185ff348c7ba33fd8f7/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4b61720243e76813094ed0013ab238f52b29e0161b2915b295dde84ee491d73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4b61720243e76813094ed0013ab238f52b29e0161b2915b295dde84ee491d73/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4a2af27c2a8965d975bb507d67b8d7913a9892064d4c08b198143ada77b09ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4a2af27c2a8965d975bb507d67b8d7913a9892064d4c08b198143ada77b09ca/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4b0e1a6cb16382af5ca734b2db6c94add203f2465348d2dbdf1c85142a6166d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4b0e1a6cb16382af5ca734b2db6c94add203f2465348d2dbdf1c85142a6166d/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da6e4c556c6db14a784e659952326a07ea2ec68109b5372ada31741260f0e2a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da6e4c556c6db14a784e659952326a07ea2ec68109b5372ada31741260f0e2a9/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e31931c322e80c8afa97b10b1cd548b0d128ba65167ac7e1d53d2125ce55169d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e31931c322e80c8afa97b10b1cd548b0d128ba65167ac7e1d53d2125ce55169d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f05728b61827da09eea34c3b104724b4cd12c729fd64e7332ce5068feeb7315a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f05728b61827da09eea34c3b104724b4cd12c729fd64e7332ce5068feeb7315a/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f239d4a549679d7a053791231a64d4299843f672b22739f51676b76b35ec965a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f239d4a549679d7a053791231a64d4299843f672b22739f51676b76b35ec965a/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f40874cd434631d0c82ea59ab46ad95fd6e64bc4acc4c886497ca4e636478f2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f40874cd434631d0c82ea59ab46ad95fd6e64bc4acc4c886497ca4e636478f2b/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fec64e24bba00eece98e1de08b20b8fed1d37aed0f5bf0467d654ebb87336389/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fec64e24bba00eece98e1de08b20b8fed1d37aed0f5bf0467d654ebb87336389/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b74c6c-f107-4795-bfbf-a7437c10918e/volumes/kubernetes.io~projected/kube-api-access-gz2zb:{mountpoint:/var/lib/kubelet/pods/03b74c6c-f107-4795-bfbf-a7437c10918e/volumes/kubernetes.io~projected/kube-api-access-gz2zb major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b74c6c-f107-4795-bfbf-a7437c10918e/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/03b74c6c-f107-4795-bfbf-a7437c10918e/volumes/kubernetes.io~secret/signing-key major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/068756c4-d0b9-4002-beb5-7661990b1145/volumes/kubernetes.io~projected/kube-api-access-jqvrb:{mountpoint:/var/lib/kubelet/pods/068756c4-d0b9-4002-beb5-7661990b1145/volumes/kubernetes.io~projected/kube-api-access-jqvrb major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0945eba8-6898-4dfe-ae35-687496c3b596/volumes/kubernetes.io~projected/kube-api-access-5n2jd:{mountpoint:/var/lib/kubelet/pods/0945eba8-6898-4dfe-ae35-687496c3b596/volumes/kubernetes.io~projected/kube-api-access-5n2jd major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bbacc63-2772-4fa1-a868-afc6bc41fa44/volumes/kubernetes.io~projected/kube-api-access-5lz4w:{mountpoint:/var/lib/kubelet/pods/0bbacc63-2772-4fa1-a868-afc6bc41fa44/volumes/kubernetes.io~projected/kube-api-access-5lz4w major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dbdd6e7-a8de-4990-9aec-358480af8521/volumes/kubernetes.io~projected/kube-api-access-dsx67:{mountpoint:/var/lib/kubelet/pods/0dbdd6e7-a8de-4990-9aec-358480af8521/volumes/kubernetes.io~projected/kube-api-access-dsx67 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dbdd6e7-a8de-4990-9aec-358480af8521/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/0dbdd6e7-a8de-4990-9aec-358480af8521/volumes/kubernetes.io~secret/default-certificate major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dbdd6e7-a8de-4990-9aec-358480af8521/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/0dbdd6e7-a8de-4990-9aec-358480af8521/volumes/kubernetes.io~secret/stats-auth major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1dd4ba4e-edba-4732-9f72-54c83b78d6fc/volumes/kubernetes.io~projected/kube-api-access-s8pjb:{mountpoint:/var/lib/kubelet/pods/1dd4ba4e-edba-4732-9f72-54c83b78d6fc/volumes/kubernetes.io~projected/kube-api-access-s8pjb major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/volumes/kubernetes.io~projected/kube-api-access-m8ltn:{mountpoint:/var/lib/kubelet/pods/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/volumes/kubernetes.io~projected/kube-api-access-m8ltn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29923b63-cb6e-4a44-b098-ebd0317c1c14/volumes/kubernetes.io~projected/kube-api-access-w5bzs:{mountpoint:/var/lib/kubelet/pods/29923b63-cb6e-4a44-b098-ebd0317c1c14/volumes/kubernetes.io~projected/kube-api-access-w5bzs major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29923b63-cb6e-4a44-b098-ebd0317c1c14/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/29923b63-cb6e-4a44-b098-ebd0317c1c14/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35d6f6db-753a-4d1e-ae56-58659a5baab1/volumes/kubernetes.io~projected/kube-api-access-jgl8k:{mountpoint:/var/lib/kubelet/pods/35d6f6db-753a-4d1e-ae56-58659a5baab1/volumes/kubernetes.io~projected/kube-api-access-jgl8k major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35dabb24-0583-4de3-9a5d-21c8ef04c7c5/volumes/kubernetes.io~projected/kube-api-access-kmgwz:{mountpoint:/var/lib/kubelet/pods/35dabb24-0583-4de3-9a5d-21c8ef04c7c5/volumes/kubernetes.io~projected/kube-api-access-kmgwz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~projected/kube-api-access-hlzh9:{mountpoint:/var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~projected/kube-api-access-hlzh9 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e7c1aa6-a7c5-4ed6-a871-43a9bf568693/volumes/kubernetes.io~projected/kube-api-access-ffflv:{mountpoint:/var/lib/kubelet/pods/3e7c1aa6-a7c5-4ed6-a871-43a9bf568693/volumes/kubernetes.io~projected/kube-api-access-ffflv major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40609c09-1682-4c49-8eae-af9760072fc5/volumes/kubernetes.io~projected/kube-api-access-l7x5g:{mountpoint:/var/lib/kubelet/pods/40609c09-1682-4c49-8eae-af9760072fc5/volumes/kubernetes.io~projected/kube-api-access-l7x5g major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8/volumes/kubernetes.io~projected/kube-api-access-jmls2:{mountpoint:/var/lib/kubelet/pods/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8/volumes/kubernetes.io~projected/kube-api-access-jmls2 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8/volumes/kubernetes.io~secret/cert major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56eddc7b-b1d4-404d-8d08-e482801db81c/volumes/kubernetes.io~projected/kube-api-access-qmgz9:{mountpoint:/var/lib/kubelet/pods/56eddc7b-b1d4-404d-8d08-e482801db81c/volumes/kubernetes.io~projected/kube-api-access-qmgz9 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/578bb9eb-99cf-4506-ae47-6d23defa622a/volumes/kubernetes.io~projected/kube-api-access-9zjxv:{mountpoint:/var/lib/kubelet/pods/578bb9eb-99cf-4506-ae47-6d23defa622a/volumes/kubernetes.io~projected/kube-api-access-9zjxv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a9404d9-1240-451f-8064-255583bb5da6/volumes/kubernetes.io~projected/kube-api-access-2pd85:{mountpoint:/var/lib/kubelet/pods/5a9404d9-1240-451f-8064-255583bb5da6/volumes/kubernetes.io~projected/kube-api-access-2pd85 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f7b33fa-69cf-4582-a4b4-844ed91e2e26/volumes/kubernetes.io~projected/kube-api-access-m8j4x:{mountpoint:/var/lib/kubelet/pods/5f7b33fa-69cf-4582-a4b4-844ed91e2e26/volumes/kubernetes.io~projected/kube-api-access-m8j4x major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f7b33fa-69cf-4582-a4b4-844ed91e2e26/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5f7b33fa-69cf-4582-a4b4-844ed91e2e26/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69dd95c4-b504-46d3-97a9-ac864aef3f85/volumes/kubernetes.io~projected/kube-api-access-wbgkf:{mountpoint:/var/lib/kubelet/pods/69dd95c4-b504-46d3-97a9-ac864aef3f85/volumes/kubernetes.io~projected/kube-api-access-wbgkf major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e537c04-9b2a-4d85-9ffd-db08c554c838/volumes/kubernetes.io~projected/kube-api-access-ss6gx:{mountpoint:/var/lib/kubelet/pods/7e537c04-9b2a-4d85-9ffd-db08c554c838/volumes/kubernetes.io~projected/kube-api-access-ss6gx major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bac6548-3f94-4861-84ef-5bf4bfd57409/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9bac6548-3f94-4861-84ef-5bf4bfd57409/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aedffb88-1606-439b-968d-ab12c5a2b984/volumes/kubernetes.io~projected/kube-api-access-vlpxw:{mountpoint:/var/lib/kubelet/pods/aedffb88-1606-439b-968d-ab12c5a2b984/volumes/kubernetes.io~projected/kube-api-access-vlpxw major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aedffb88-1606-439b-968d-ab12c5a2b984/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/aedffb88-1606-439b-968d-ab12c5a2b984/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b86c6b47-2b14-4151-a04a-12c1505e928a/volumes/kubernetes.io~projected/kube-api-access-47qwf:{mountpoint:/var/lib/kubelet/pods/b86c6b47-2b14-4151-a04a-12c1505e928a/volumes/kubernetes.io~projected/kube-api-access-47qwf major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba906d22-0572-4a25-b444-09b5c553825d/volumes/kubernetes.io~projected/kube-api-access-m5v2x:{mountpoint:/var/lib/kubelet/pods/ba906d22-0572-4a25-b444-09b5c553825d/volumes/kubernetes.io~projected/kube-api-access-m5v2x major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2661da3-c179-4adf-888b-56e2a89b5a02/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c2661da3-c179-4adf-888b-56e2a89b5a02/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2661da3-c179-4adf-888b-56e2a89b5a02/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c2661da3-c179-4adf-888b-56e2a89b5a02/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2661da3-c179-4adf-888b-56e2a89b5a02/volumes/kubernetes.io~projected/kube-api-access-2kxzl:{mountpoint:/var/lib/kubelet/pods/c2661da3-c179-4adf-888b-56e2a89b5a02/volumes/kubernetes.io~projected/kube-api-access-2kxzl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~projected/kube-api-access-8qk2c:{mountpoint:/var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~projected/kube-api-access-8qk2c major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce877b3e-cadd-4cf5-afe7-620e5251c1e3/volumes/kubernetes.io~projected/kube-api-access-z2sdh:{mountpoint:/var/lib/kubelet/pods/ce877b3e-cadd-4cf5-afe7-620e5251c1e3/volumes/kubernetes.io~projected/kube-api-access-z2sdh major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb92e08-27a4-458a-818b-0215acd6dbd5/volumes/kubernetes.io~projected/kube-api-access-ff57n:{mountpoint:/var/lib/kubelet/pods/dcb92e08-27a4-458a-818b-0215acd6dbd5/volumes/kubernetes.io~projected/kube-api-access-ff57n major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de30627e-073b-4b99-9ef3-93dc856e4be6/volumes/kubernetes.io~projected/kube-api-access-4ps6p:{mountpoint:/var/lib/kubelet/pods/de30627e-073b-4b99-9ef3-93dc856e4be6/volumes/kubernetes.io~projected/kube-api-access-4ps6p major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e86d4545-6769-4133-ba67-674e17ab3392/volumes/kubernetes.io~projected/kube-api-access-xmsxs:{mountpoint:/var/lib/kubelet/pods/e86d4545-6769-4133-ba67-674e17ab3392/volumes/kubernetes.io~projected/kube-api-access-xmsxs major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8bf626d-af0f-47d8-8d61-686c390139af/volumes/kubernetes.io~projected/kube-api-access-xs7gc:{mountpoint:/var/lib/kubelet/pods/e8bf626d-af0f-47d8-8d61-686c390139af/volumes/kubernetes.io~projected/kube-api-access-xs7gc major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8bf626d-af0f-47d8-8d61-686c390139af/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e8bf626d-af0f-47d8-8d61-686c390139af/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed900688-4264-4ba8-8daa-83e717a6f13f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ed900688-4264-4ba8-8daa-83e717a6f13f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8ea16e9-1545-4881-b7c2-883d8b09a6fe/volumes/kubernetes.io~projected/kube-api-access-wz8vn:{mountpoint:/var/lib/kubelet/pods/f8ea16e9-1545-4881-b7c2-883d8b09a6fe/volumes/kubernetes.io~projected/kube-api-access-wz8vn major:0 minor:234 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/659a3eede05cc02491c1deb5a886421780670cab7f717fba9db909107cab54b9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6ce23bb111ac927e11d85c3c873e09e497ad0fe5061f38f77953301d96a66bd7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a8e12b1c5312b2b0d19e40996a4cfe14c955d87d3a45e9052aa10ef2b2a19d21/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/8487452b8e91e64d52dd444b9c58af705e9b45b716f06db40fa7bb1bf8d12173/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/94a83fa3187b9a3bd612a88a2e2ef9a1e94583083d04a6d36299ed8876666969/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0d6c68d32af2855da6f8975c78b015768e3c32bd0f483c0be25dd8b7f6409b23/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/77cd33523c6f5f9494426d43fcbb3641e5a304326aafc364c76bc6ebb8e5c458/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/76c639ab49d329eb4e0e7a2c104351d10dc4c566ddf543906ee49f7b754e36c1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/aa991c0d6045bd848174b6f519af5b87a942bbe305df770b498ec8ca531ffaeb/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/3b4de1bdff4d648cdf0418383d48699238cba4b49a66f8d2339ac893a67557bd/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f8501571316ae8983f7327b3b19cade60e499f913b80f8ae5698a8eb1cd3b529/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/d2868850bff1af40b469060a4f50ef7294dc33ca87a1f0d800030b66af54c11f/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8a619d5a25fc02331f7a40c23586f3c05189c94b44bd6e6482375d39e9584827/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/48a171e27184417e7983757feef90f87aa1006202bd7ad07ed36106a2da62f2c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/40e11765db761fd6a059fabe6e4eba0650608005476b4ae099c8a643fba5edc7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c5d82b709449c0cbb9845de3b339b55b3b2d2435af5a0b48e16724e0fcf25c44/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a37e063816d460905d78bb9a9a26e3fa63df4975462a72791ee49dd3c860a1b9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9f5e933605987e7a34797cd7502ff07dca0c683a238bc24a4183fe25df90ffd1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f56006c1fc31902a9fc7192891f3f831b14142c1aad652aa0e17d442c1746828/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/8ffe28d5360db6bfaef85be589e0e416390b6dea7318bc1080c1797bdb68af66/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c9090bd569369dee07242761fa308d4698a87fb9d868733b895fea22939007d1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/33ecfe618acb2fd7f6357f66299b4d6c21748e085cc475c6a7dfde0197c4ab32/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/966e03f9714709223966cfbba2696be6613e6fb75684ba6bffb40bcf44672ca0/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/a8c3d8f532c44fd354c986bd3719cb42d9f8b24d7b3ba2632478fbcbe5245bb7/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/83612f312396148646400d34c03c2a49970d8eef986072beb5803dd44dbcbe00/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/6c462f9fd47f5a8cd94b6e716c97a192a5a2e4621bfd6f8160136f3c8f79e91c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d7d71a67e7f1485057edc48a20ebd56abdd443415238e447e10eefdad871d720/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/ea135efdb17804195faeb0e1f85d708ba587a142d3d05a0bbb71ff68ec59667b/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/8a743ea5a8bc1c402c499f6200b406926caf9c71a9da9be6e9bc6bbbb0930cf3/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/c181e60e28887fdb9fafc73d8095a48ea48188e6ab0ae91ff32a66171f6bee53/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/508e45f680722f11b7ad10e86010f0ec15584e95195fcf83a189fb2ff65ec9fb/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/f8d2a2a4b0dbca6d93e72ececfda78ad4032b4df109ae0360c543099d02dcbb3/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/5200f84e7d631b103a98e9d9af76f58c66492ae3fd2cad7a894baccc51a300b6/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/93e6ea3bf66a7a524038cda521d32195f3f1e067e3f76a2eab968669c6985cb2/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/58081c2f60a55c665a64c2efae9ec39ce7961c376e4efb5e6541f413140a15e5/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/c311ea4195803a8a3f220e452523f9feadec0f38f979c870dae8c315313a8666/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d40f0cffc56e751c97c0e5a72a596568093762cb186e66779490a62f951975fe/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/5ddcf5bd64eb27e69eccdc44961a8fefa29cad3e979e1d0f0a39264a3ac11494/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/a46123232e806dee55de68647fe43accdac19819ecfec8da198eae44c97ea481/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/8c2994b1346ed3dc2740e908311baf2c65ccffb9b1b3175dac7a84c9e562bfed/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/95bc6c439fcc1f32b488e5be92c494ba026ec2dc098fe3ee347beb3be8c02dfa/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/a7a31e05ae96db42e22055ea9829fe7126a24eab99828012a247781c20f2bb67/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/39a911820674a9d98e52b1f1473175ec11f14b318b53f161399d7d01404c745a/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/a6e523f6152678d93bbb82016b39b20bc595f0a6f424ccf56fafa41c92b23c2a/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/4587c28226c4d5b89d57b98631b6f470ffbb065c0d610f1639ff2136b76e1fcf/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/5f810ed92ac4a22f390737217e57d72613fa5ba06ca50804ff22649281e70f52/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/b2df480a519e21207a0543424f343e903d513170e9b636b8d9573957e82cf660/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/b2ddf9d1b5281ae7d4ff1f916e19fa030c2e49dcd964a1f87cfe3de4346240fd/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/ba5e2fd592542f52aca0f4b93165bb2db1d7b21db5938891debb7deccb54ef2f/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/bfd2ed8012e2ec2021a1c2620759f17d7b1f3ec1f5aad98f755eee5221456de6/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/24f4809966d5414fd9f7df50dbf24d2c2120d465259884b13e5f38e3db799e00/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/7dfcf57642cc14e9058e090361511759403794a6b482dbc7ec55c09758773227/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5d61d40308815bea29ee5049717e1728a9c1f5a95ed3457bdef183a729a93a98/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/010c2691d8d7d5a015bebfcac63cb00d766a345f8a34bd5e7b37333b6bef1e4c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/cc7bf15d300c75c490bc0156ac3e68f36868be59e15268ae4bdf202ff93fb2a5/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/25f00344c1d308387af633866bd24791c6e6b906dfb903a6e7463e08148c6e2d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0d81f5ee267eaebe068731ad0838ff804f60149c3ce558bd7c458734dbf6ccb7/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b50d8581132a7921e9a70a6377f609446f8b94743b2a4013943d6be5ee07b78b/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/915aac33e566d6f77fd5282db414d7421e5ba1e43ffe71a42ea42c4f5250821d/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/5aae820ad26ee47f8637adea5acfaa10af932b7bd6cf3b8fcd25c1fbd6a6ca93/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b9a64e596a6315f9efb71570844f1b1d324a760dc61b38b442b826766a7d75c2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4300c9060fcfe5a0eee4e4b62e3944fc7423b24dc365faceb7a0aaf41a7a8a50/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 10:28:42.110888 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.110522 5086 manager.go:217] Machine: {Timestamp:2026-11-03 10:28:42.110033289 +0000 UTC m=+0.094904563 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112773 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217080005f15520b9fb04d257e3093 SystemUUID:ec217080-005f-1552-0b9f-b04d257e3093 BootID:b2a24977-1059-42e4-ad97-35a7c5e7d5aa Filesystems:[{Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0dbdd6e7-a8de-4990-9aec-358480af8521/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71eff809985da4d6fcb78437170620f5a5f97bb544ab28f0ba74970ef67200c4/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b35e39ed7b164fda5ca6fb8cd155503010d6e252f0ad4185ff348c7ba33fd8f7/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aedffb88-1606-439b-968d-ab12c5a2b984/volumes/kubernetes.io~projected/kube-api-access-vlpxw DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ca00b168f15cd54b49d1dbea606cfc42ceb5ec7c15efb6b109bae1ff1bcb4b3/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dbdd6e7-a8de-4990-9aec-358480af8521/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e7c1aa6-a7c5-4ed6-a871-43a9bf568693/volumes/kubernetes.io~projected/kube-api-access-ffflv DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/56eddc7b-b1d4-404d-8d08-e482801db81c/volumes/kubernetes.io~projected/kube-api-access-qmgz9 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aedffb88-1606-439b-968d-ab12c5a2b984/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/03b74c6c-f107-4795-bfbf-a7437c10918e/volumes/kubernetes.io~projected/kube-api-access-gz2zb DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/03b74c6c-f107-4795-bfbf-a7437c10918e/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:336 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/578bb9eb-99cf-4506-ae47-6d23defa622a/volumes/kubernetes.io~projected/kube-api-access-9zjxv DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b6ccf6ff952fbc103bcd1e6f526caecc7523ce358ac57838d3ab7995167f13b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4b0e1a6cb16382af5ca734b2db6c94add203f2465348d2dbdf1c85142a6166d/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62620f06b1463de681bce804b51273054ea05e1c04fbf9b8bcd0b40d52b2b8e6/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c2661da3-c179-4adf-888b-56e2a89b5a02/volumes/kubernetes.io~projected/kube-api-access-2kxzl DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da6e4c556c6db14a784e659952326a07ea2ec68109b5372ada31741260f0e2a9/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/051e792f9023c91f8519b920f269eb39e1f6a1199e8fdc0c4d1a142f9f9b26c2/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e537c04-9b2a-4d85-9ffd-db08c554c838/volumes/kubernetes.io~projected/kube-api-access-ss6gx DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0dbdd6e7-a8de-4990-9aec-358480af8521/volumes/kubernetes.io~projected/kube-api-access-dsx67 DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40609c09-1682-4c49-8eae-af9760072fc5/volumes/kubernetes.io~projected/kube-api-access-l7x5g DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/068756c4-d0b9-4002-beb5-7661990b1145/volumes/kubernetes.io~projected/kube-api-access-jqvrb DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~projected/kube-api-access-8qk2c DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95765fea9b25353b77dc1b85ad2509a2b7987d01d6407e960614d2a1972fb0d3/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14c5e03cffb3143d08863d35eef13abdaecc9ea44b96338a4c7dcfc99e4c0244/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35d6f6db-753a-4d1e-ae56-58659a5baab1/volumes/kubernetes.io~projected/kube-api-access-jgl8k DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/596e649f74a93fff5772be92eac7e59f0cd8d95345040fce7338e9e66435da4c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bac6548-3f94-4861-84ef-5bf4bfd57409/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e31931c322e80c8afa97b10b1cd548b0d128ba65167ac7e1d53d2125ce55169d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce877b3e-cadd-4cf5-afe7-620e5251c1e3/volumes/kubernetes.io~projected/kube-api-access-z2sdh DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f7b33fa-69cf-4582-a4b4-844ed91e2e26/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e86d4545-6769-4133-ba67-674e17ab3392/volumes/kubernetes.io~projected/kube-api-access-xmsxs DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed900688-4264-4ba8-8daa-83e717a6f13f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a9404d9-1240-451f-8064-255583bb5da6/volumes/kubernetes.io~projected/kube-api-access-2pd85 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f8ea16e9-1545-4881-b7c2-883d8b09a6fe/volumes/kubernetes.io~projected/kube-api-access-wz8vn DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/69dd95c4-b504-46d3-97a9-ac864aef3f85/volumes/kubernetes.io~projected/kube-api-access-wbgkf DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e448fcae41e1f65a439d0cefbd87e6bf0d76f7f785a5a3591fafcf7e0e4bbc4/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8bf626d-af0f-47d8-8d61-686c390139af/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29923b63-cb6e-4a44-b098-ebd0317c1c14/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4a2af27c2a8965d975bb507d67b8d7913a9892064d4c08b198143ada77b09ca/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4597f40640b48da624e13186a05078f486220e826d4ed2e16cef82684f15f8a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8bf626d-af0f-47d8-8d61-686c390139af/volumes/kubernetes.io~projected/kube-api-access-xs7gc DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f7b33fa-69cf-4582-a4b4-844ed91e2e26/volumes/kubernetes.io~projected/kube-api-access-m8j4x DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29923b63-cb6e-4a44-b098-ebd0317c1c14/volumes/kubernetes.io~projected/kube-api-access-w5bzs DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38d40c28f8ac7cf12e57b4eebd78c95fbf271d61abd27cba2cea3e82a2759dbc/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0945eba8-6898-4dfe-ae35-687496c3b596/volumes/kubernetes.io~projected/kube-api-access-5n2jd DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c2661da3-c179-4adf-888b-56e2a89b5a02/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~projected/kube-api-access-hlzh9 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1dd4ba4e-edba-4732-9f72-54c83b78d6fc/volumes/kubernetes.io~projected/kube-api-access-s8pjb DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2da30282695c66f194a8923a7abcdf27b0a6f3ee2028ed120ae5f6bc04141445/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35dabb24-0583-4de3-9a5d-21c8ef04c7c5/volumes/kubernetes.io~projected/kube-api-access-kmgwz DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/885a845459902eae25f1863fe55ea16efd70b3aae005cd200792d5535322a573/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53b6aeee6f808d6fb6ea455381b1fbb98c8f2fd329357c9c440415b319b9688c/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8/volumes/kubernetes.io~projected/kube-api-access-jmls2 DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/volumes/kubernetes.io~projected/kube-api-access-m8ltn DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c2661da3-c179-4adf-888b-56e2a89b5a02/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bbacc63-2772-4fa1-a868-afc6bc41fa44/volumes/kubernetes.io~projected/kube-api-access-5lz4w DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4b61720243e76813094ed0013ab238f52b29e0161b2915b295dde84ee491d73/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de30627e-073b-4b99-9ef3-93dc856e4be6/volumes/kubernetes.io~projected/kube-api-access-4ps6p DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f05728b61827da09eea34c3b104724b4cd12c729fd64e7332ce5068feeb7315a/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f239d4a549679d7a053791231a64d4299843f672b22739f51676b76b35ec965a/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95b9d82a4dac6860777ab8889652042c7ac53c87baaa71d1c0e696d1aea4105d/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fec64e24bba00eece98e1de08b20b8fed1d37aed0f5bf0467d654ebb87336389/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f40874cd434631d0c82ea59ab46ad95fd6e64bc4acc4c886497ca4e636478f2b/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/642a45164a990d98917febf0ee367018f01c3675a855436e5f2a675434835b9b/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba906d22-0572-4a25-b444-09b5c553825d/volumes/kubernetes.io~projected/kube-api-access-m5v2x DeviceMajor:0 DeviceMinor:333 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b86c6b47-2b14-4151-a04a-12c1505e928a/volumes/kubernetes.io~projected/kube-api-access-47qwf DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb92e08-27a4-458a-818b-0215acd6dbd5/volumes/kubernetes.io~projected/kube-api-access-ff57n DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:14c5e03cffb3143 MacAddress:36:13:55:05:2b:ea Speed:10000 Mtu:8901} {Name:2da30282695c66f MacAddress:4e:02:72:6c:9c:6c Speed:10000 Mtu:8901} {Name:38d40c28f8ac7cf MacAddress:52:94:a8:84:cb:a5 Speed:10000 Mtu:8901} {Name:3ca00b168f15cd5 MacAddress:46:a7:39:df:b9:36 Speed:10000 Mtu:8901} {Name:53b6aeee6f808d6 MacAddress:3a:68:20:5c:a7:f1 Speed:10000 Mtu:8901} {Name:62620f06b1463de MacAddress:26:ba:ee:a7:2a:30 Speed:10000 Mtu:8901} {Name:642a45164a990d9 MacAddress:da:99:69:68:02:07 Speed:10000 Mtu:8901} {Name:6b6ccf6ff952fbc MacAddress:a2:55:b2:1a:89:2c Speed:10000 Mtu:8901} {Name:95765fea9b25353 MacAddress:2e:32:f3:ba:61:69 Speed:10000 Mtu:8901} {Name:95b9d82a4dac686 MacAddress:e2:53:a7:88:2c:64 Speed:10000 Mtu:8901} {Name:b4b61720243e768 MacAddress:7a:43:87:32:97:f9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e2:d2:12:ef Speed:0 Mtu:9001} {Name:br-int MacAddress:66:b2:3c:27:7d:11 Speed:0 Mtu:8901} {Name:c4a2af27c2a8965 MacAddress:56:8e:7b:f2:13:65 Speed:10000 Mtu:8901} {Name:c4b0e1a6cb16382 MacAddress:06:3d:27:14:d6:da Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e2:d2:12:ef Speed:0 Mtu:9001} {Name:f239d4a549679d7 MacAddress:66:a8:5b:d1:23:8b Speed:10000 Mtu:8901} {Name:f40874cd434631d MacAddress:e6:98:55:32:62:86 Speed:10000 Mtu:8901} {Name:fec64e24bba00ee MacAddress:9e:97:75:bf:8b:03 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9e:c5:3e:6e:24:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 10:28:42.111108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.110893 5086 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 10:28:42.111108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.110939 5086 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 10:28:42.111270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111245 5086 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 10:28:42.111433 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111271 5086 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-90.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 10:28:42.111481 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111437 5086 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 10:28:42.111481 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111445 5086 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 10:28:42.111481 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111455 5086 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 10:28:42.111481 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111475 5086 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 10:28:42.111588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111496 5086 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 10:28:42.111698 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111689 5086 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 10:28:42.111832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111823 5086 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 10:28:42.111872 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111862 5086 kubelet.go:493] "Attempting to sync node with API server" Nov 03 10:28:42.111916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111876 5086 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 10:28:42.111916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111886 5086 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 10:28:42.111916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111896 5086 kubelet.go:405] "Adding apiserver pod source" Nov 03 10:28:42.111916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.111904 5086 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 10:28:42.113042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.113021 5086 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 10:28:42.113042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.113039 5086 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 10:28:42.113671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.113654 5086 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.116750 5086 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.116768 5086 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.116900 5086 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117307 5086 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117352 5086 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117639 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117668 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117675 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117681 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117687 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117694 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117701 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117706 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117719 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117726 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 10:28:42.117953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.117739 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 10:28:42.118591 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.118449 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 10:28:42.118591 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.118496 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 10:28:42.118591 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.118506 5086 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 10:28:42.122687 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.122670 5086 server.go:1263] "Started kubelet" Nov 03 10:28:42.122793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.122725 5086 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 10:28:42.122885 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.122840 5086 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 10:28:42.122937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.122908 5086 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 10:28:42.123213 ip-10-0-139-90 systemd[1]: Started Kubernetes Kubelet. Nov 03 10:28:42.124265 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.123456 5086 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 10:28:42.124265 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.124150 5086 server.go:317] "Adding debug handlers to kubelet server" Nov 03 10:28:42.126325 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.126289 5086 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 10:28:42.126461 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.126329 5086 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 10:28:42.126461 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.126390 5086 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 10:23:02 +0000 UTC" deadline="2028-06-19 09:37:42.527501961 +0000 UTC" Nov 03 10:28:42.126461 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.126423 5086 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14255h9m0.401081704s" Nov 03 10:28:42.126629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.126482 5086 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 10:28:42.126629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.126493 5086 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 10:28:42.126629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.126613 5086 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 10:28:42.129324 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:42.126875 5086 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" Nov 03 10:28:42.129324 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.127799 5086 factory.go:55] Registering systemd factory Nov 03 10:28:42.129324 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.127816 5086 factory.go:223] Registration of the systemd container factory successfully Nov 03 10:28:42.132183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.131940 5086 factory.go:153] Registering CRI-O factory Nov 03 10:28:42.132183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.131964 5086 factory.go:223] Registration of the crio container factory successfully Nov 03 10:28:42.132183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.132046 5086 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 10:28:42.132183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.132067 5086 factory.go:103] Registering Raw factory Nov 03 10:28:42.132183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.132093 5086 manager.go:1199] Started watching for new ooms in manager Nov 03 10:28:42.132485 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.132459 5086 manager.go:319] Starting recovery of all containers Nov 03 10:28:42.138894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.138843 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf" volumeName="kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-config" seLinuxMountContext="" Nov 03 10:28:42.138992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.138901 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2661da3-c179-4adf-888b-56e2a89b5a02" volumeName="kubernetes.io/projected/c2661da3-c179-4adf-888b-56e2a89b5a02-kube-api-access-2kxzl" seLinuxMountContext="" Nov 03 10:28:42.138992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.138921 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb00dfc2-12f0-4fc3-ba37-0bc83b344602" volumeName="kubernetes.io/configmap/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-nginx-conf" seLinuxMountContext="" Nov 03 10:28:42.138992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.138934 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bbacc63-2772-4fa1-a868-afc6bc41fa44" volumeName="kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w" seLinuxMountContext="" Nov 03 10:28:42.138992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.138949 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e7c1aa6-a7c5-4ed6-a871-43a9bf568693" volumeName="kubernetes.io/projected/3e7c1aa6-a7c5-4ed6-a871-43a9bf568693-kube-api-access-ffflv" seLinuxMountContext="" Nov 03 10:28:42.138992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.138963 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56eddc7b-b1d4-404d-8d08-e482801db81c" volumeName="kubernetes.io/projected/56eddc7b-b1d4-404d-8d08-e482801db81c-kube-api-access-qmgz9" seLinuxMountContext="" Nov 03 10:28:42.138992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.138976 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedffb88-1606-439b-968d-ab12c5a2b984" volumeName="kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-config" seLinuxMountContext="" Nov 03 10:28:42.138992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.138988 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce877b3e-cadd-4cf5-afe7-620e5251c1e3" volumeName="kubernetes.io/projected/ce877b3e-cadd-4cf5-afe7-620e5251c1e3-kube-api-access-z2sdh" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139003 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8ea16e9-1545-4881-b7c2-883d8b09a6fe" volumeName="kubernetes.io/configmap/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-config-volume" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139015 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8" volumeName="kubernetes.io/projected/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-kube-api-access-jmls2" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139045 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578bb9eb-99cf-4506-ae47-6d23defa622a" volumeName="kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139059 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2661da3-c179-4adf-888b-56e2a89b5a02" volumeName="kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-tmp" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139073 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="068756c4-d0b9-4002-beb5-7661990b1145" volumeName="kubernetes.io/projected/068756c4-d0b9-4002-beb5-7661990b1145-kube-api-access-jqvrb" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139085 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35d6f6db-753a-4d1e-ae56-58659a5baab1" volumeName="kubernetes.io/projected/35d6f6db-753a-4d1e-ae56-58659a5baab1-kube-api-access-jgl8k" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139100 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="369e7292-b645-4eb6-9a5d-62e08910f1ef" volumeName="kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139112 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c634e6ef-e187-4409-a0a0-bae5793bc137" volumeName="kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139127 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0945eba8-6898-4dfe-ae35-687496c3b596" volumeName="kubernetes.io/projected/0945eba8-6898-4dfe-ae35-687496c3b596-kube-api-access-5n2jd" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139140 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dbdd6e7-a8de-4990-9aec-358480af8521" volumeName="kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-default-certificate" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139154 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29923b63-cb6e-4a44-b098-ebd0317c1c14" volumeName="kubernetes.io/secret/29923b63-cb6e-4a44-b098-ebd0317c1c14-serving-cert" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139172 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e86d4545-6769-4133-ba67-674e17ab3392" volumeName="kubernetes.io/projected/e86d4545-6769-4133-ba67-674e17ab3392-kube-api-access-xmsxs" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139184 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35dabb24-0583-4de3-9a5d-21c8ef04c7c5" volumeName="kubernetes.io/configmap/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-serviceca" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139196 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56eddc7b-b1d4-404d-8d08-e482801db81c" volumeName="kubernetes.io/configmap/56eddc7b-b1d4-404d-8d08-e482801db81c-iptables-alerter-script" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139210 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578bb9eb-99cf-4506-ae47-6d23defa622a" volumeName="kubernetes.io/projected/578bb9eb-99cf-4506-ae47-6d23defa622a-kube-api-access-9zjxv" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139223 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a9404d9-1240-451f-8064-255583bb5da6" volumeName="kubernetes.io/configmap/5a9404d9-1240-451f-8064-255583bb5da6-telemetry-config" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139235 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f7b33fa-69cf-4582-a4b4-844ed91e2e26" volumeName="kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-snapshots" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139247 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e537c04-9b2a-4d85-9ffd-db08c554c838" volumeName="kubernetes.io/configmap/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-config" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139272 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c634e6ef-e187-4409-a0a0-bae5793bc137" volumeName="kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139286 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de30627e-073b-4b99-9ef3-93dc856e4be6" volumeName="kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-daemon-config" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139321 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b74c6c-f107-4795-bfbf-a7437c10918e" volumeName="kubernetes.io/projected/03b74c6c-f107-4795-bfbf-a7437c10918e-kube-api-access-gz2zb" seLinuxMountContext="" Nov 03 10:28:42.139343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139344 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35d6f6db-753a-4d1e-ae56-58659a5baab1" volumeName="kubernetes.io/empty-dir/35d6f6db-753a-4d1e-ae56-58659a5baab1-tmp-dir" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139356 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40609c09-1682-4c49-8eae-af9760072fc5" volumeName="kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139368 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2661da3-c179-4adf-888b-56e2a89b5a02" volumeName="kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-tuned" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139380 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de30627e-073b-4b99-9ef3-93dc856e4be6" volumeName="kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-cni-binary-copy" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139392 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de30627e-073b-4b99-9ef3-93dc856e4be6" volumeName="kubernetes.io/projected/de30627e-073b-4b99-9ef3-93dc856e4be6-kube-api-access-4ps6p" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139408 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8ea16e9-1545-4881-b7c2-883d8b09a6fe" volumeName="kubernetes.io/empty-dir/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-tmp-dir" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139420 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8ea16e9-1545-4881-b7c2-883d8b09a6fe" volumeName="kubernetes.io/projected/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-kube-api-access-wz8vn" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139434 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf" volumeName="kubernetes.io/secret/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139446 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="369e7292-b645-4eb6-9a5d-62e08910f1ef" volumeName="kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139458 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f7b33fa-69cf-4582-a4b4-844ed91e2e26" volumeName="kubernetes.io/projected/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-kube-api-access-m8j4x" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139470 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bac6548-3f94-4861-84ef-5bf4bfd57409" volumeName="kubernetes.io/secret/9bac6548-3f94-4861-84ef-5bf4bfd57409-agent-certs" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139484 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c634e6ef-e187-4409-a0a0-bae5793bc137" volumeName="kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139496 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b74c6c-f107-4795-bfbf-a7437c10918e" volumeName="kubernetes.io/configmap/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-cabundle" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139507 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dbdd6e7-a8de-4990-9aec-358480af8521" volumeName="kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-stats-auth" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139521 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf" volumeName="kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-env-overrides" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139534 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29923b63-cb6e-4a44-b098-ebd0317c1c14" volumeName="kubernetes.io/configmap/29923b63-cb6e-4a44-b098-ebd0317c1c14-config" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139546 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="369e7292-b645-4eb6-9a5d-62e08910f1ef" volumeName="kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139560 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e537c04-9b2a-4d85-9ffd-db08c554c838" volumeName="kubernetes.io/projected/7e537c04-9b2a-4d85-9ffd-db08c554c838-kube-api-access-ss6gx" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139573 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedffb88-1606-439b-968d-ab12c5a2b984" volumeName="kubernetes.io/secret/aedffb88-1606-439b-968d-ab12c5a2b984-serving-cert" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139586 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="369e7292-b645-4eb6-9a5d-62e08910f1ef" volumeName="kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139597 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578bb9eb-99cf-4506-ae47-6d23defa622a" volumeName="kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-binary-copy" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139610 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69dd95c4-b504-46d3-97a9-ac864aef3f85" volumeName="kubernetes.io/projected/69dd95c4-b504-46d3-97a9-ac864aef3f85-kube-api-access-wbgkf" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139624 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedffb88-1606-439b-968d-ab12c5a2b984" volumeName="kubernetes.io/projected/aedffb88-1606-439b-968d-ab12c5a2b984-kube-api-access-vlpxw" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139637 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bbacc63-2772-4fa1-a868-afc6bc41fa44" volumeName="kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139649 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedffb88-1606-439b-968d-ab12c5a2b984" volumeName="kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-trusted-ca" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139662 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c634e6ef-e187-4409-a0a0-bae5793bc137" volumeName="kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139674 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8bf626d-af0f-47d8-8d61-686c390139af" volumeName="kubernetes.io/secret/e8bf626d-af0f-47d8-8d61-686c390139af-serving-cert" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139696 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f7b33fa-69cf-4582-a4b4-844ed91e2e26" volumeName="kubernetes.io/secret/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-serving-cert" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139709 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="369e7292-b645-4eb6-9a5d-62e08910f1ef" volumeName="kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139712 5086 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139722 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf" volumeName="kubernetes.io/projected/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-kube-api-access-m8ltn" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139747 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35dabb24-0583-4de3-9a5d-21c8ef04c7c5" volumeName="kubernetes.io/projected/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-kube-api-access-kmgwz" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139759 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="369e7292-b645-4eb6-9a5d-62e08910f1ef" volumeName="kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139773 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f7b33fa-69cf-4582-a4b4-844ed91e2e26" volumeName="kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-service-ca-bundle" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139787 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0945eba8-6898-4dfe-ae35-687496c3b596" volumeName="kubernetes.io/empty-dir/0945eba8-6898-4dfe-ae35-687496c3b596-data-volume" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139803 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c634e6ef-e187-4409-a0a0-bae5793bc137" volumeName="kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139817 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8bf626d-af0f-47d8-8d61-686c390139af" volumeName="kubernetes.io/configmap/e8bf626d-af0f-47d8-8d61-686c390139af-config" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139831 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed900688-4264-4ba8-8daa-83e717a6f13f" volumeName="kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139844 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bbacc63-2772-4fa1-a868-afc6bc41fa44" volumeName="kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139856 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf" volumeName="kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-script-lib" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139869 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29923b63-cb6e-4a44-b098-ebd0317c1c14" volumeName="kubernetes.io/projected/29923b63-cb6e-4a44-b098-ebd0317c1c14-kube-api-access-w5bzs" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139884 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8" volumeName="kubernetes.io/secret/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-cert" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139897 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f7b33fa-69cf-4582-a4b4-844ed91e2e26" volumeName="kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-tmp" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139908 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b86c6b47-2b14-4151-a04a-12c1505e928a" volumeName="kubernetes.io/projected/b86c6b47-2b14-4151-a04a-12c1505e928a-kube-api-access-47qwf" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139920 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba906d22-0572-4a25-b444-09b5c553825d" volumeName="kubernetes.io/projected/ba906d22-0572-4a25-b444-09b5c553825d-kube-api-access-m5v2x" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139957 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c634e6ef-e187-4409-a0a0-bae5793bc137" volumeName="kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139971 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bbacc63-2772-4fa1-a868-afc6bc41fa44" volumeName="kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.139984 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dbdd6e7-a8de-4990-9aec-358480af8521" volumeName="kubernetes.io/projected/0dbdd6e7-a8de-4990-9aec-358480af8521-kube-api-access-dsx67" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140000 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578bb9eb-99cf-4506-ae47-6d23defa622a" volumeName="kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140012 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a9404d9-1240-451f-8064-255583bb5da6" volumeName="kubernetes.io/projected/5a9404d9-1240-451f-8064-255583bb5da6-kube-api-access-2pd85" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140024 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c634e6ef-e187-4409-a0a0-bae5793bc137" volumeName="kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140045 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb92e08-27a4-458a-818b-0215acd6dbd5" volumeName="kubernetes.io/projected/dcb92e08-27a4-458a-818b-0215acd6dbd5-kube-api-access-ff57n" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140053 5086 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140059 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8bf626d-af0f-47d8-8d61-686c390139af" volumeName="kubernetes.io/projected/e8bf626d-af0f-47d8-8d61-686c390139af-kube-api-access-xs7gc" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140153 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b74c6c-f107-4795-bfbf-a7437c10918e" volumeName="kubernetes.io/secret/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-key" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140175 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dd4ba4e-edba-4732-9f72-54c83b78d6fc" volumeName="kubernetes.io/projected/1dd4ba4e-edba-4732-9f72-54c83b78d6fc-kube-api-access-s8pjb" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140191 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="369e7292-b645-4eb6-9a5d-62e08910f1ef" volumeName="kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140204 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f7b33fa-69cf-4582-a4b4-844ed91e2e26" volumeName="kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-trusted-ca-bundle" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140216 5086 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bac6548-3f94-4861-84ef-5bf4bfd57409" volumeName="kubernetes.io/configmap/9bac6548-3f94-4861-84ef-5bf4bfd57409-konnectivity-ca" seLinuxMountContext="" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140229 5086 reconstruct.go:97] "Volume reconstruction finished" Nov 03 10:28:42.140290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.140236 5086 reconciler.go:29] "Reconciler: start to sync state" Nov 03 10:28:42.143773 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.143754 5086 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 10:28:42.152672 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.152650 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:28:42.153134 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.153115 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:28:42.161056 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.161013 5086 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 10:28:42.162422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.162395 5086 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 10:28:42.162422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.162422 5086 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 10:28:42.162604 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.162447 5086 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 10:28:42.162604 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:42.162489 5086 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 10:28:42.166844 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.166823 5086 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:42.167283 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.167261 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:28:42.167370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.167267 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:28:42.169847 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.169808 5086 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="f93396fdfa90c3007afd4364169a5373ca700d4e6c18b58976ad53864d4fc6b6" exitCode=0 Nov 03 10:28:42.169847 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.169848 5086 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="114eb62506484ad1cc984a38ea27de31dafd024844f12f3957446f1739da9f02" exitCode=0 Nov 03 10:28:42.170002 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.169857 5086 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="c9e55ce8bd53a868b073b9b25ed501dbe33e783bab43d5329f55439ea684b6f2" exitCode=0 Nov 03 10:28:42.170002 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.169865 5086 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="d672a3649217b21958464cb9abfd26e182a502e8ebde05730b44856e97e10762" exitCode=0 Nov 03 10:28:42.170002 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.169873 5086 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="ed7ead42173cf7c10d9d35228bb8f11fc08a2252b7727c365edbbe45c4af3db9" exitCode=0 Nov 03 10:28:42.170002 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.169880 5086 generic.go:356] "Generic (PLEG): container finished" podID="578bb9eb-99cf-4506-ae47-6d23defa622a" containerID="e9f300155bbc73989b028d0818c27d201554c31703b1201be8ca942a5a54b48b" exitCode=0 Nov 03 10:28:42.172006 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.171985 5086 generic.go:356] "Generic (PLEG): container finished" podID="29fda1c39c64a6789fa2412aae915cee" containerID="9cf8497ea538d7f17a1b347e73765398c544143d5bf3f8a675a51a5fc1285b2d" exitCode=0 Nov 03 10:28:42.177799 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.177781 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:28:42.177887 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.177809 5086 generic.go:356] "Generic (PLEG): container finished" podID="aedffb88-1606-439b-968d-ab12c5a2b984" containerID="47d119010ab7ed8dc33e90f86b5fa1777f750bfedc319ffb5c7860275ab6f22c" exitCode=255 Nov 03 10:28:42.206674 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.206600 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:28:42.207121 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.207092 5086 generic.go:356] "Generic (PLEG): container finished" podID="27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf" containerID="ff756edfb62d4f45e84b4f6669014581c405c1d7c8286e6c8231812cd2f39356" exitCode=1 Nov 03 10:28:42.222311 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.222273 5086 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 10:28:42.238759 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.238739 5086 manager.go:324] Recovery completed Nov 03 10:28:42.260577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260553 5086 cpu_manager.go:226] "Starting" policy="none" Nov 03 10:28:42.260577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260572 5086 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 10:28:42.260762 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260593 5086 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 10:28:42.260762 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260741 5086 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 10:28:42.260850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260756 5086 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 10:28:42.260850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260804 5086 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 10:28:42.260850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260811 5086 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 10:28:42.260850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260818 5086 policy_none.go:50] "Start" Nov 03 10:28:42.260850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260828 5086 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 10:28:42.260850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260838 5086 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 10:28:42.261128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260973 5086 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 10:28:42.261128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260980 5086 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 10:28:42.261128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.260986 5086 policy_none.go:44] "Start" Nov 03 10:28:42.262605 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:42.262580 5086 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 10:28:42.267460 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.267426 5086 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 10:28:42.270672 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.270657 5086 manager.go:341] "Starting Device Plugin manager" Nov 03 10:28:42.271180 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:42.271165 5086 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 10:28:42.271180 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.271180 5086 server.go:87] "Starting device plugin registration server" Nov 03 10:28:42.271460 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.271450 5086 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 10:28:42.271494 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.271462 5086 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 10:28:42.271615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.271587 5086 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 10:28:42.271720 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.271685 5086 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 10:28:42.271802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.271729 5086 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 10:28:42.271802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.271737 5086 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 10:28:42.273422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.272886 5086 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T10:28:42.271719187Z","UUID":"662ad819-3ce4-410c-8fb3-6b5a120505f6","Handler":null,"Name":"","Endpoint":""} Nov 03 10:28:42.273422 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:42.273039 5086 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 10:28:42.274246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.274228 5086 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 10:28:42.274342 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.274267 5086 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 10:28:42.371613 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.371579 5086 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 10:28:42.372978 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.372961 5086 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 10:28:42.373058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.372996 5086 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 10:28:42.373058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.373008 5086 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 10:28:42.373058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.373050 5086 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.386445 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.386423 5086 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.386513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.386496 5086 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.463472 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.463359 5086 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"] Nov 03 10:28:42.464190 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.464173 5086 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.464257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.464163 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerStarted","Data":"a4b32acd22a0544b22d6d9126734e31ffd51bc3aae458560149756074246bfd5"} Nov 03 10:28:42.464257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.464212 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerDied","Data":"9cf8497ea538d7f17a1b347e73765398c544143d5bf3f8a675a51a5fc1285b2d"} Nov 03 10:28:42.464257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.464231 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerStarted","Data":"a4597f40640b48da624e13186a05078f486220e826d4ed2e16cef82684f15f8a"} Nov 03 10:28:42.464386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.464274 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"bb77e2dfdbc4a16f3bcede022ac475a7","Type":"ContainerStarted","Data":"09d235f1fb0be41d80b42ce7e1ae36c709bfbc1e02e6d43dccbd29f7b60ae16d"} Nov 03 10:28:42.464386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.464310 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"bb77e2dfdbc4a16f3bcede022ac475a7","Type":"ContainerStarted","Data":"71eff809985da4d6fcb78437170620f5a5f97bb544ab28f0ba74970ef67200c4"} Nov 03 10:28:42.464441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.464407 5086 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.474864 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.474844 5086 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 10:28:42.475004 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:42.474889 5086 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.475368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.475358 5086 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 10:28:42.475405 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:42.475382 5086 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.545778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.545741 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.545778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.545781 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.545974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.545802 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.646379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.646338 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.646379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.646380 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.646584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.646408 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.646584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.646447 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.646584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.646474 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:42.646584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:42.646453 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:43.113160 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.113122 5086 apiserver.go:52] "Watching apiserver" Nov 03 10:28:43.123565 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.123539 5086 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 10:28:43.124276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.124249 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-dgk2v","openshift-cluster-node-tuning-operator/tuned-r877m","openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f","kube-system/global-pull-secret-syncer-4vttz","openshift-network-console/networking-console-plugin-86c478bb88-h65vg","openshift-insights/periodic-gathering-2j5wq-fjfdd","openshift-multus/network-metrics-daemon-ztmr2","openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf","openshift-network-diagnostics/network-check-target-t8sqq","openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw","openshift-ovn-kubernetes/ovnkube-node-fq8sx","openshift-ingress-canary/ingress-canary-jhr4s","openshift-multus/multus-additional-cni-plugins-z57kl","openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb","openshift-dns/node-resolver-jhp98","kube-system/konnectivity-agent-x9pz8","openshift-ingress/router-default-788447d7cc-tl5jd","openshift-image-registry/image-registry-675dcb4c7d-4p5mj","openshift-service-ca/service-ca-5fffc97bdf-4wzkm","openshift-dns/dns-default-fb26v","openshift-console-operator/console-operator-7cd56b8dbc-2m2xr","openshift-insights/insights-operator-7c4986d449-8rp5q","openshift-insights/insights-runtime-extractor-rhvqw","openshift-image-registry/image-registry-678cf4bb67-v4rz9","kube-system/kas-connection-checker-685895c7c9-wk475","kube-system/kas-connection-checker-685895c7c9-q22k4","openshift-multus/multus-z7d6k","openshift-image-registry/node-ca-ssnbk","openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk","kube-system/kas-connection-checker-685895c7c9-qkwqb"] Nov 03 10:28:43.126356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.126332 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:43.127118 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.127094 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:43.128736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.128711 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.128859 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.128732 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.130587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.130520 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.131220 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.131196 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.131374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.131356 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.131475 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.131461 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 10:28:43.131822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.131623 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mh9pr\"" Nov 03 10:28:43.131822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.131726 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 10:28:43.131822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.131782 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.131855 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.131953 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fhcv2\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.131979 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.132359 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.132499 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.132506 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2smdw\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.132596 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k5ll2\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.132805 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.132858 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.132960 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.133042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.132977 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.133818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133164 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 10:28:43.133818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133242 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 10:28:43.133818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133329 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.133818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133346 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.133818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133483 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.133818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133599 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 10:28:43.133818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133616 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8kqg8\"" Nov 03 10:28:43.133818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133759 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.134233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133884 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xm5kr\"" Nov 03 10:28:43.134233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.133936 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.134721 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.134704 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.135703 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.135684 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 10:28:43.135836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.135823 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:43.138437 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.138407 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 10:28:43.138602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.138580 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 10:28:43.140209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.140190 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.140209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.140202 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.140362 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.140228 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.140362 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.140323 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.140511 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.140492 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z6b7k\"" Nov 03 10:28:43.140574 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.140520 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 10:28:43.140626 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.140594 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7khc5\"" Nov 03 10:28:43.140701 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.140681 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.141964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.141946 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 10:28:43.142035 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.141975 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 10:28:43.142201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.142184 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 10:28:43.142268 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.142251 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 10:28:43.142268 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.142260 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 10:28:43.142455 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.142436 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m29gn\"" Nov 03 10:28:43.142549 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.142531 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.142884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.142870 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:43.144398 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.144379 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.145317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.145280 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.145884 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.145903 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.145910 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.145946 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.145966 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-58tzx\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.146004 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.146020 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.146037 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lq8vs\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.146046 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.146085 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.146171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.146150 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.146803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.146213 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.146803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.145971 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 10:28:43.146803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.146309 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 10:28:43.146803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.146431 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wbpkj\"" Nov 03 10:28:43.147098 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.147075 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.147098 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.147087 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fb6wm\"" Nov 03 10:28:43.147356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.147334 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 10:28:43.147445 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.147369 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 10:28:43.147445 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.147422 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 10:28:43.150522 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150483 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:43.150614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150539 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:43.150614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150577 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/5a9404d9-1240-451f-8064-255583bb5da6-kube-api-access-2pd85\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:43.150727 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150617 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgkf\" (UniqueName: \"kubernetes.io/projected/69dd95c4-b504-46d3-97a9-ac864aef3f85-kube-api-access-wbgkf\") pod \"kas-connection-checker-685895c7c9-q22k4\" (UID: \"69dd95c4-b504-46d3-97a9-ac864aef3f85\") " pod="kube-system/kas-connection-checker-685895c7c9-q22k4" Nov 03 10:28:43.150727 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150715 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qwf\" (UniqueName: \"kubernetes.io/projected/b86c6b47-2b14-4151-a04a-12c1505e928a-kube-api-access-47qwf\") pod \"volume-data-source-validator-58c9f5f46d-h8sdw\" (UID: \"b86c6b47-2b14-4151-a04a-12c1505e928a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" Nov 03 10:28:43.150821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150743 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6gx\" (UniqueName: \"kubernetes.io/projected/7e537c04-9b2a-4d85-9ffd-db08c554c838-kube-api-access-ss6gx\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:43.150821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150785 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:43.150950 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150932 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7gc\" (UniqueName: \"kubernetes.io/projected/e8bf626d-af0f-47d8-8d61-686c390139af-kube-api-access-xs7gc\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:43.150997 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150947 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:43.150997 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.150973 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8pjb\" (UniqueName: \"kubernetes.io/projected/1dd4ba4e-edba-4732-9f72-54c83b78d6fc-kube-api-access-s8pjb\") pod \"kas-connection-checker-685895c7c9-qkwqb\" (UID: \"1dd4ba4e-edba-4732-9f72-54c83b78d6fc\") " pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" Nov 03 10:28:43.151124 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151013 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29923b63-cb6e-4a44-b098-ebd0317c1c14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:43.151124 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151092 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5a9404d9-1240-451f-8064-255583bb5da6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:43.151219 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151136 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bzs\" (UniqueName: \"kubernetes.io/projected/29923b63-cb6e-4a44-b098-ebd0317c1c14-kube-api-access-w5bzs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:43.151219 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151176 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8bf626d-af0f-47d8-8d61-686c390139af-serving-cert\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:43.151219 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151210 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffflv\" (UniqueName: \"kubernetes.io/projected/3e7c1aa6-a7c5-4ed6-a871-43a9bf568693-kube-api-access-ffflv\") pod \"kas-connection-checker-685895c7c9-wk475\" (UID: \"3e7c1aa6-a7c5-4ed6-a871-43a9bf568693\") " pod="kube-system/kas-connection-checker-685895c7c9-wk475" Nov 03 10:28:43.151375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151251 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29923b63-cb6e-4a44-b098-ebd0317c1c14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:43.151375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151288 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8bf626d-af0f-47d8-8d61-686c390139af-config\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:43.151375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151326 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29923b63-cb6e-4a44-b098-ebd0317c1c14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:43.151537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151518 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8bf626d-af0f-47d8-8d61-686c390139af-config\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:43.151817 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.151799 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5a9404d9-1240-451f-8064-255583bb5da6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:43.152657 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.152630 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29923b63-cb6e-4a44-b098-ebd0317c1c14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:43.152736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.152637 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8bf626d-af0f-47d8-8d61-686c390139af-serving-cert\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:43.152822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.152805 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.158367 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.158343 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/5a9404d9-1240-451f-8064-255583bb5da6-kube-api-access-2pd85\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:43.158953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.158927 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8pjb\" (UniqueName: \"kubernetes.io/projected/1dd4ba4e-edba-4732-9f72-54c83b78d6fc-kube-api-access-s8pjb\") pod \"kas-connection-checker-685895c7c9-qkwqb\" (UID: \"1dd4ba4e-edba-4732-9f72-54c83b78d6fc\") " pod="kube-system/kas-connection-checker-685895c7c9-qkwqb" Nov 03 10:28:43.159039 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.158992 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgkf\" (UniqueName: \"kubernetes.io/projected/69dd95c4-b504-46d3-97a9-ac864aef3f85-kube-api-access-wbgkf\") pod \"kas-connection-checker-685895c7c9-q22k4\" (UID: \"69dd95c4-b504-46d3-97a9-ac864aef3f85\") " pod="kube-system/kas-connection-checker-685895c7c9-q22k4" Nov 03 10:28:43.159106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.159084 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bzs\" (UniqueName: \"kubernetes.io/projected/29923b63-cb6e-4a44-b098-ebd0317c1c14-kube-api-access-w5bzs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f\" (UID: \"29923b63-cb6e-4a44-b098-ebd0317c1c14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" Nov 03 10:28:43.159195 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.159178 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7gc\" (UniqueName: \"kubernetes.io/projected/e8bf626d-af0f-47d8-8d61-686c390139af-kube-api-access-xs7gc\") pod \"service-ca-operator-544d749c8c-t4kgk\" (UID: \"e8bf626d-af0f-47d8-8d61-686c390139af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" Nov 03 10:28:43.159428 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.159383 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffflv\" (UniqueName: \"kubernetes.io/projected/3e7c1aa6-a7c5-4ed6-a871-43a9bf568693-kube-api-access-ffflv\") pod \"kas-connection-checker-685895c7c9-wk475\" (UID: \"3e7c1aa6-a7c5-4ed6-a871-43a9bf568693\") " pod="kube-system/kas-connection-checker-685895c7c9-wk475" Nov 03 10:28:43.159582 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.159559 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qwf\" (UniqueName: \"kubernetes.io/projected/b86c6b47-2b14-4151-a04a-12c1505e928a-kube-api-access-47qwf\") pod \"volume-data-source-validator-58c9f5f46d-h8sdw\" (UID: \"b86c6b47-2b14-4151-a04a-12c1505e928a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8sdw" Nov 03 10:28:43.159694 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.159677 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6gx\" (UniqueName: \"kubernetes.io/projected/7e537c04-9b2a-4d85-9ffd-db08c554c838-kube-api-access-ss6gx\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:43.221564 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.221528 5086 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:43.221711 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.221591 5086 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:43.228451 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.228428 5086 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 10:28:43.229200 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229170 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229220 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229249 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229220 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229248 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229289 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229329 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229340 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229343 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229346 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229374 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229378 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.229457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229443 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 10:28:43.230100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229495 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 10:28:43.230100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229592 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.230100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229608 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 10:28:43.230100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229631 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.230100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229642 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.230100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229643 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 10:28:43.230100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229658 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 10:28:43.230100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229698 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:28:43.230100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.229726 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.230515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.230105 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 10:28:43.230515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.230148 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 10:28:43.230515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.230398 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 10:28:43.231365 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231342 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5hs5j\"" Nov 03 10:28:43.231441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231350 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-khtn4\"" Nov 03 10:28:43.231441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231387 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 10:28:43.231441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231404 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 10:28:43.231441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231420 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pthxs\"" Nov 03 10:28:43.231441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231435 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 10:28:43.231441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231448 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-trm8g\"" Nov 03 10:28:43.231662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231462 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zcr5r\"" Nov 03 10:28:43.231662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231484 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xjpbn\"" Nov 03 10:28:43.231844 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231822 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ssvsh\"" Nov 03 10:28:43.231949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231847 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4c7q2\"" Nov 03 10:28:43.231949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.231898 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 10:28:43.232015 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232000 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5cc62\"" Nov 03 10:28:43.232112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232099 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jkrmv\"" Nov 03 10:28:43.232183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232168 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w9nrb\"" Nov 03 10:28:43.232251 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232238 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nxhw\"" Nov 03 10:28:43.232291 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232281 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6fzr5\"" Nov 03 10:28:43.232344 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232307 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 10:28:43.232388 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232378 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 10:28:43.232461 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232449 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6w9j8\"" Nov 03 10:28:43.232511 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232451 5086 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 10:28:43.232556 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.232541 5086 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:43.232587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232557 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qqtl6\"" Nov 03 10:28:43.232651 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.232638 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 10:28:43.233197 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.233185 5086 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 10:28:43.233238 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.233214 5086 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" Nov 03 10:28:43.251580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251552 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.251726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251594 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8j4x\" (UniqueName: \"kubernetes.io/projected/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-kube-api-access-m8j4x\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.251726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251617 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-socket-dir-parent\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.251726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251638 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0945eba8-6898-4dfe-ae35-687496c3b596-crio-socket\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.251726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251661 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.251863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251737 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-system-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.251863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251812 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-bin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.251863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251853 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-netd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.251958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251899 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-socket-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.251958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251925 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-dbus\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:43.251958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251948 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.252107 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.251953 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-conf-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.252107 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252001 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9bac6548-3f94-4861-84ef-5bf4bfd57409-agent-certs\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:43.252107 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252044 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.252107 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252084 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.252278 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252108 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-systemd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.252278 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252127 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz8vn\" (UniqueName: \"kubernetes.io/projected/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-kube-api-access-wz8vn\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.252278 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252143 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.252278 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252144 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.252278 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252188 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.252278 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252222 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-multus-certs\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.252278 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252246 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ltn\" (UniqueName: \"kubernetes.io/projected/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-kube-api-access-m8ltn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252290 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252311 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-kubernetes\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252352 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252376 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0945eba8-6898-4dfe-ae35-687496c3b596-data-volume\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252405 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-netns\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252430 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252470 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252491 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0945eba8-6898-4dfe-ae35-687496c3b596-data-volume\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252493 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9bac6548-3f94-4861-84ef-5bf4bfd57409-agent-certs\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252510 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-etc-kubernetes\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252553 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-var-lib-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.252596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252587 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-serviceca\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252627 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252658 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-cert\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252676 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252736 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252775 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-systemd\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252807 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-tmp\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252848 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-cert\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252846 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-serviceca\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252868 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252905 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d6f6db-753a-4d1e-ae56-58659a5baab1-tmp-dir\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252919 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-tmp\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252931 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-cni-binary-copy\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252969 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252995 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d6f6db-753a-4d1e-ae56-58659a5baab1-tmp-dir\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.252998 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.253036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253038 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-multus\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253059 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253065 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-hostroot\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253099 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253146 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253161 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253174 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253210 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253212 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-cni-binary-copy\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253271 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253361 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253408 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysconfig\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253501 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9bac6548-3f94-4861-84ef-5bf4bfd57409-konnectivity-ca\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253532 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-config\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:43.253546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253548 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253599 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253629 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-netns\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253685 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9bac6548-3f94-4861-84ef-5bf4bfd57409-konnectivity-ca\") pod \"konnectivity-agent-x9pz8\" (UID: \"9bac6548-3f94-4861-84ef-5bf4bfd57409\") " pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253669 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjxv\" (UniqueName: \"kubernetes.io/projected/578bb9eb-99cf-4506-ae47-6d23defa622a-kube-api-access-9zjxv\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253723 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-run\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253732 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-config\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253771 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5v2x\" (UniqueName: \"kubernetes.io/projected/ba906d22-0572-4a25-b444-09b5c553825d-kube-api-access-m5v2x\") pod \"migrator-66df5d496b-xfbpb\" (UID: \"ba906d22-0572-4a25-b444-09b5c553825d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253820 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-device-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253851 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-k8s-cni-cncf-io\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253877 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253912 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.253968 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-config-volume\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254003 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254027 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254061 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aedffb88-1606-439b-968d-ab12c5a2b984-serving-cert\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:43.254112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254087 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254144 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qk2c\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254159 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-config-volume\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254206 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvrb\" (UniqueName: \"kubernetes.io/projected/068756c4-d0b9-4002-beb5-7661990b1145-kube-api-access-jqvrb\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254259 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254260 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-lib-modules\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254308 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aedffb88-1606-439b-968d-ab12c5a2b984-serving-cert\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254320 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-default-certificate\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254354 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-bin\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254381 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgwz\" (UniqueName: \"kubernetes.io/projected/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-kube-api-access-kmgwz\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254412 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-cnibin\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254450 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254543 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-default-certificate\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254551 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254582 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-etc-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254617 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254645 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-tmp-dir\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254718 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254725 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-tmp-dir\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254740 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-service-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254756 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-serving-cert\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254777 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmgz9\" (UniqueName: \"kubernetes.io/projected/56eddc7b-b1d4-404d-8d08-e482801db81c-kube-api-access-qmgz9\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:43.254780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254794 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254817 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-tuned\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254843 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254860 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254886 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-tuned\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254900 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmsxs\" (UniqueName: \"kubernetes.io/projected/e86d4545-6769-4133-ba67-674e17ab3392-kube-api-access-xmsxs\") pod \"network-check-source-6f78d59f99-wkrtf\" (UID: \"e86d4545-6769-4133-ba67-674e17ab3392\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254927 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254953 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.254976 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2zb\" (UniqueName: \"kubernetes.io/projected/03b74c6c-f107-4795-bfbf-a7437c10918e-kube-api-access-gz2zb\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255006 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255032 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-sys-fs\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255046 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255059 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-cabundle\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255077 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-serving-cert\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255119 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255162 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-host\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.255163 5086 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255191 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-var-lib-kubelet\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255232 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff57n\" (UniqueName: \"kubernetes.io/projected/dcb92e08-27a4-458a-818b-0215acd6dbd5-kube-api-access-ff57n\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255235 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-service-ca-bundle\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.255247 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls podName:5a9404d9-1240-451f-8064-255583bb5da6 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:43.755227713 +0000 UTC m=+1.740098998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rchbs" (UID: "5a9404d9-1240-451f-8064-255583bb5da6") : secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255280 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-cabundle\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255404 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-snapshots\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.255629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255461 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255816 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255832 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-snapshots\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255850 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aedffb88-1606-439b-968d-ab12c5a2b984-trusted-ca\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255893 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-binary-copy\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.255935 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35d6f6db-753a-4d1e-ae56-58659a5baab1-hosts-file\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256085 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgl8k\" (UniqueName: \"kubernetes.io/projected/35d6f6db-753a-4d1e-ae56-58659a5baab1-kube-api-access-jgl8k\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256125 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56eddc7b-b1d4-404d-8d08-e482801db81c-host-slash\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256155 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-script-lib\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256179 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-cni-binary-copy\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256188 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpxw\" (UniqueName: \"kubernetes.io/projected/aedffb88-1606-439b-968d-ab12c5a2b984-kube-api-access-vlpxw\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256221 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-daemon-config\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256260 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56eddc7b-b1d4-404d-8d08-e482801db81c-iptables-alerter-script\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256264 5086 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256310 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lz4w\" (UniqueName: \"kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256365 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-sys\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256399 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmls2\" (UniqueName: \"kubernetes.io/projected/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-kube-api-access-jmls2\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256434 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:43.256634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256641 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-nginx-conf\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:43.257414 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.256964 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-daemon-config\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.257414 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257224 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/56eddc7b-b1d4-404d-8d08-e482801db81c-iptables-alerter-script\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:43.257599 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257573 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-script-lib\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.257730 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257618 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:43.257730 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257638 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.257730 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257661 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:43.257730 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257704 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-tmp\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.257730 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257732 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsx67\" (UniqueName: \"kubernetes.io/projected/0dbdd6e7-a8de-4990-9aec-358480af8521-kube-api-access-dsx67\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.257987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257750 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-kubelet\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.257987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257777 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-node-log\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.257987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257807 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-system-cni-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.257987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257838 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-key\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:43.257987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257859 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:43.257987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257876 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-cnibin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.257987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257912 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-kubelet\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.257987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.257941 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-ovn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.258406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258077 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-conf\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.258462 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258440 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxzl\" (UniqueName: \"kubernetes.io/projected/c2661da3-c179-4adf-888b-56e2a89b5a02-kube-api-access-2kxzl\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.258527 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258487 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-registration-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.258586 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258527 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2sdh\" (UniqueName: \"kubernetes.io/projected/ce877b3e-cadd-4cf5-afe7-620e5251c1e3-kube-api-access-z2sdh\") pod \"migrator-66df5d496b-lf8q6\" (UID: \"ce877b3e-cadd-4cf5-afe7-620e5251c1e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" Nov 03 10:28:43.258586 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258575 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03b74c6c-f107-4795-bfbf-a7437c10918e-signing-key\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:43.258735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258646 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ps6p\" (UniqueName: \"kubernetes.io/projected/de30627e-073b-4b99-9ef3-93dc856e4be6-kube-api-access-4ps6p\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.258735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258682 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-stats-auth\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.258735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258721 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/578bb9eb-99cf-4506-ae47-6d23defa622a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.258890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258794 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovn-node-metrics-cert\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.258890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.258820 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.259152 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259129 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-kubelet-config\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:43.259229 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259186 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n2jd\" (UniqueName: \"kubernetes.io/projected/0945eba8-6898-4dfe-ae35-687496c3b596-kube-api-access-5n2jd\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.259229 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259139 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovn-node-metrics-cert\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.259368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259226 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed900688-4264-4ba8-8daa-83e717a6f13f-original-pull-secret\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:43.259368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259269 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-log-socket\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.259368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259280 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.259368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259322 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-env-overrides\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.259588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259473 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-tmp\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.259588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259485 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-stats-auth\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.259588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259514 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-modprobe-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.259588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259551 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-host\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.259787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259595 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzh9\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.259787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259621 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-systemd-units\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.259787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259641 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-slash\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.259787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259682 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-env-overrides\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.259787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.259737 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-config\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.260127 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.260110 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:43.260190 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.260128 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-ovnkube-config\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.261058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.261037 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e537c04-9b2a-4d85-9ffd-db08c554c838-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69g9h\" (UID: \"7e537c04-9b2a-4d85-9ffd-db08c554c838\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:43.262905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.262531 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz8vn\" (UniqueName: \"kubernetes.io/projected/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-kube-api-access-wz8vn\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.262905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.262734 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8j4x\" (UniqueName: \"kubernetes.io/projected/5f7b33fa-69cf-4582-a4b4-844ed91e2e26-kube-api-access-m8j4x\") pod \"insights-operator-7c4986d449-8rp5q\" (UID: \"5f7b33fa-69cf-4582-a4b4-844ed91e2e26\") " pod="openshift-insights/insights-operator-7c4986d449-8rp5q" Nov 03 10:28:43.263106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.263088 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5v2x\" (UniqueName: \"kubernetes.io/projected/ba906d22-0572-4a25-b444-09b5c553825d-kube-api-access-m5v2x\") pod \"migrator-66df5d496b-xfbpb\" (UID: \"ba906d22-0572-4a25-b444-09b5c553825d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xfbpb" Nov 03 10:28:43.264134 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.264115 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2zb\" (UniqueName: \"kubernetes.io/projected/03b74c6c-f107-4795-bfbf-a7437c10918e-kube-api-access-gz2zb\") pod \"service-ca-5fffc97bdf-4wzkm\" (UID: \"03b74c6c-f107-4795-bfbf-a7437c10918e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4wzkm" Nov 03 10:28:43.264342 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.264321 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmsxs\" (UniqueName: \"kubernetes.io/projected/e86d4545-6769-4133-ba67-674e17ab3392-kube-api-access-xmsxs\") pod \"network-check-source-6f78d59f99-wkrtf\" (UID: \"e86d4545-6769-4133-ba67-674e17ab3392\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wkrtf" Nov 03 10:28:43.265504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.264965 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjxv\" (UniqueName: \"kubernetes.io/projected/578bb9eb-99cf-4506-ae47-6d23defa622a-kube-api-access-9zjxv\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.265504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.265088 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.265504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.265096 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsx67\" (UniqueName: \"kubernetes.io/projected/0dbdd6e7-a8de-4990-9aec-358480af8521-kube-api-access-dsx67\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.265504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.265092 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.265504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.265099 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff57n\" (UniqueName: \"kubernetes.io/projected/dcb92e08-27a4-458a-818b-0215acd6dbd5-kube-api-access-ff57n\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:43.265504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.265274 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpxw\" (UniqueName: \"kubernetes.io/projected/aedffb88-1606-439b-968d-ab12c5a2b984-kube-api-access-vlpxw\") pod \"console-operator-7cd56b8dbc-2m2xr\" (UID: \"aedffb88-1606-439b-968d-ab12c5a2b984\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:43.265504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.265336 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7x5g\" (UniqueName: \"kubernetes.io/projected/40609c09-1682-4c49-8eae-af9760072fc5-kube-api-access-l7x5g\") pod \"network-check-target-t8sqq\" (UID: \"40609c09-1682-4c49-8eae-af9760072fc5\") " pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:43.265504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.265466 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmgz9\" (UniqueName: \"kubernetes.io/projected/56eddc7b-b1d4-404d-8d08-e482801db81c-kube-api-access-qmgz9\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:43.265879 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.265620 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qk2c\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.265879 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.265629 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lz4w\" (UniqueName: \"kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w\") pod \"periodic-gathering-2j5wq-fjfdd\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:28:43.266158 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.266133 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmls2\" (UniqueName: \"kubernetes.io/projected/4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8-kube-api-access-jmls2\") pod \"ingress-canary-jhr4s\" (UID: \"4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8\") " pod="openshift-ingress-canary/ingress-canary-jhr4s" Nov 03 10:28:43.266251 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.266142 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgl8k\" (UniqueName: \"kubernetes.io/projected/35d6f6db-753a-4d1e-ae56-58659a5baab1-kube-api-access-jgl8k\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:43.266432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.266411 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvrb\" (UniqueName: \"kubernetes.io/projected/068756c4-d0b9-4002-beb5-7661990b1145-kube-api-access-jqvrb\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.266592 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.266575 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ltn\" (UniqueName: \"kubernetes.io/projected/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-kube-api-access-m8ltn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.266653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.266595 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgwz\" (UniqueName: \"kubernetes.io/projected/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-kube-api-access-kmgwz\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:43.269098 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.269078 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2sdh\" (UniqueName: \"kubernetes.io/projected/ce877b3e-cadd-4cf5-afe7-620e5251c1e3-kube-api-access-z2sdh\") pod \"migrator-66df5d496b-lf8q6\" (UID: \"ce877b3e-cadd-4cf5-afe7-620e5251c1e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lf8q6" Nov 03 10:28:43.270658 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.270642 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n2jd\" (UniqueName: \"kubernetes.io/projected/0945eba8-6898-4dfe-ae35-687496c3b596-kube-api-access-5n2jd\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.270996 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.270980 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxzl\" (UniqueName: \"kubernetes.io/projected/c2661da3-c179-4adf-888b-56e2a89b5a02-kube-api-access-2kxzl\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.271665 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.271650 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzh9\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.272933 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.272916 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ps6p\" (UniqueName: \"kubernetes.io/projected/de30627e-073b-4b99-9ef3-93dc856e4be6-kube-api-access-4ps6p\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.360531 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360492 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-conf\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.360531 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360531 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-registration-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360556 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-kubelet-config\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360580 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-log-socket\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360607 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-modprobe-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360628 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-host\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360641 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-registration-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360652 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-systemd-units\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360635 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-log-socket\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360704 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-systemd-units\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360707 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-conf\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360722 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-kubelet-config\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360705 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-host\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360757 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-modprobe-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.360766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360758 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-slash\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360789 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-slash\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360796 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-socket-dir-parent\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360814 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0945eba8-6898-4dfe-ae35-687496c3b596-crio-socket\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360833 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360855 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-socket-dir-parent\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360855 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-system-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360869 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0945eba8-6898-4dfe-ae35-687496c3b596-crio-socket\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360880 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-bin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360885 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-system-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360896 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-netd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360930 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-socket-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360946 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-dbus\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360947 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-netd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360960 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-conf-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360973 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-socket-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360980 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360930 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-bin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.360999 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-systemd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361008 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed900688-4264-4ba8-8daa-83e717a6f13f-dbus\") pod \"global-pull-secret-syncer-4vttz\" (UID: \"ed900688-4264-4ba8-8daa-83e717a6f13f\") " pod="kube-system/global-pull-secret-syncer-4vttz" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361013 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-conf-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361033 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361047 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-systemd\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361058 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-multus-certs\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361083 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-kubernetes\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.361088 5086 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361105 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361122 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-netns\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361125 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-kubernetes\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361135 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-multus-certs\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361154 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-netns\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361142 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-etc-kubernetes\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361190 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-var-lib-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.361205 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:43.861184467 +0000 UTC m=+1.846055741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : secret "insights-runtime-extractor-tls" not found Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361222 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-var-lib-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.361206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361163 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-etc-kubernetes\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361241 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361270 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-systemd\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361326 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-multus\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361350 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-hostroot\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361375 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361402 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361411 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-cni-multus\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361434 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysconfig\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361423 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-hostroot\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361461 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361465 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361473 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361483 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-netns\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361488 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysconfig\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361470 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0945eba8-6898-4dfe-ae35-687496c3b596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361507 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-run\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361532 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-multus-cni-dir\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361536 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-device-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361543 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-run\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361548 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-netns\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361534 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-systemd\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361570 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-k8s-cni-cncf-io\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361600 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361623 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361574 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-device-dir\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361619 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-run-k8s-cni-cncf-io\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361656 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361692 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361725 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-etc-sysctl-d\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361724 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-lib-modules\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.361741 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:43.861725327 +0000 UTC m=+1.846596608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : configmap references non-existent config key: service-ca.crt Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361778 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-bin\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361804 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-cnibin\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361813 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-lib-modules\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361826 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361840 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-cni-bin\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361853 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361858 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-cnibin\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361877 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-etc-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361901 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361919 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361942 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-etc-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361950 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-sys-fs\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.361976 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.362011 5086 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362006 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-host\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362039 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-var-lib-kubelet\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362036 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/068756c4-d0b9-4002-beb5-7661990b1145-sys-fs\") pod \"aws-ebs-csi-driver-node-v5bzm\" (UID: \"068756c4-d0b9-4002-beb5-7661990b1145\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5bzm" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.362060 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls podName:f8ea16e9-1545-4881-b7c2-883d8b09a6fe nodeName:}" failed. No retries permitted until 2026-11-03 10:28:43.862042227 +0000 UTC m=+1.846913494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls") pod "dns-default-fb26v" (UID: "f8ea16e9-1545-4881-b7c2-883d8b09a6fe") : secret "dns-default-metrics-tls" not found Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362098 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-var-lib-kubelet\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362107 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362100 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35dabb24-0583-4de3-9a5d-21c8ef04c7c5-host\") pod \"node-ca-ssnbk\" (UID: \"35dabb24-0583-4de3-9a5d-21c8ef04c7c5\") " pod="openshift-image-registry/node-ca-ssnbk" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362138 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362169 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35d6f6db-753a-4d1e-ae56-58659a5baab1-hosts-file\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362171 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-openvswitch\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362193 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56eddc7b-b1d4-404d-8d08-e482801db81c-host-slash\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362221 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-sys\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362223 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35d6f6db-753a-4d1e-ae56-58659a5baab1-hosts-file\") pod \"node-resolver-jhp98\" (UID: \"35d6f6db-753a-4d1e-ae56-58659a5baab1\") " pod="openshift-dns/node-resolver-jhp98" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362244 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362265 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/56eddc7b-b1d4-404d-8d08-e482801db81c-host-slash\") pod \"iptables-alerter-dgk2v\" (UID: \"56eddc7b-b1d4-404d-8d08-e482801db81c\") " pod="openshift-network-operator/iptables-alerter-dgk2v" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362273 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-kubelet\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362312 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2661da3-c179-4adf-888b-56e2a89b5a02-sys\") pod \"tuned-r877m\" (UID: \"c2661da3-c179-4adf-888b-56e2a89b5a02\") " pod="openshift-cluster-node-tuning-operator/tuned-r877m" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362332 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-host-kubelet\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362417 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-node-log\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362435 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-system-cni-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362468 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-cnibin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362483 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-kubelet\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362497 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-ovn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362545 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-run-ovn\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362568 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf-node-log\") pod \"ovnkube-node-fq8sx\" (UID: \"27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362590 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/578bb9eb-99cf-4506-ae47-6d23defa622a-system-cni-dir\") pod \"multus-additional-cni-plugins-z57kl\" (UID: \"578bb9eb-99cf-4506-ae47-6d23defa622a\") " pod="openshift-multus/multus-additional-cni-plugins-z57kl" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362627 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-cnibin\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.362820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.362665 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de30627e-073b-4b99-9ef3-93dc856e4be6-host-var-lib-kubelet\") pod \"multus-z7d6k\" (UID: \"de30627e-073b-4b99-9ef3-93dc856e4be6\") " pod="openshift-multus/multus-z7d6k" Nov 03 10:28:43.365543 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.364001 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"image-registry-678cf4bb67-v4rz9\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.365543 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.364448 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb92e08-27a4-458a-818b-0215acd6dbd5-metrics-certs\") pod \"network-metrics-daemon-ztmr2\" (UID: \"dcb92e08-27a4-458a-818b-0215acd6dbd5\") " pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:43.365543 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.364587 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"image-registry-675dcb4c7d-4p5mj\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.365543 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.364599 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbdd6e7-a8de-4990-9aec-358480af8521-metrics-certs\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.365543 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.364698 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb00dfc2-12f0-4fc3-ba37-0bc83b344602-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-h65vg\" (UID: \"cb00dfc2-12f0-4fc3-ba37-0bc83b344602\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:43.426969 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.426931 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" Nov 03 10:28:43.429271 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.429251 5086 scope.go:122] "RemoveContainer" containerID="47d119010ab7ed8dc33e90f86b5fa1777f750bfedc319ffb5c7860275ab6f22c" Nov 03 10:28:43.431383 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.431368 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:43.436275 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.436260 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" Nov 03 10:28:43.444118 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.444093 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ztmr2" Nov 03 10:28:43.445105 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.445090 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:43.739994 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.739967 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h"] Nov 03 10:28:43.771345 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.771309 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:43.771517 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.771409 5086 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:43.771517 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.771474 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls podName:5a9404d9-1240-451f-8064-255583bb5da6 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:44.771451458 +0000 UTC m=+2.756322732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rchbs" (UID: "5a9404d9-1240-451f-8064-255583bb5da6") : secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:43.787525 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.787508 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:28:43.794923 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.794879 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675dcb4c7d-4p5mj"] Nov 03 10:28:43.799407 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:43.799360 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc634e6ef_e187_4409_a0a0_bae5793bc137.slice/crio-764c92b12e06a5383f9f6f0592405d4d510085cffd34fbe5ad2b4e3d0d723445 WatchSource:0}: Error finding container 764c92b12e06a5383f9f6f0592405d4d510085cffd34fbe5ad2b4e3d0d723445: Status 404 returned error can't find the container with id 764c92b12e06a5383f9f6f0592405d4d510085cffd34fbe5ad2b4e3d0d723445 Nov 03 10:28:43.812409 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.812385 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678cf4bb67-v4rz9"] Nov 03 10:28:43.816057 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:43.816016 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369e7292_b645_4eb6_9a5d_62e08910f1ef.slice/crio-cd25a392dece585f0a2ecc53e003bc66a2ca00573c4909d42c45e6c42df883a2 WatchSource:0}: Error finding container cd25a392dece585f0a2ecc53e003bc66a2ca00573c4909d42c45e6c42df883a2: Status 404 returned error can't find the container with id cd25a392dece585f0a2ecc53e003bc66a2ca00573c4909d42c45e6c42df883a2 Nov 03 10:28:43.822865 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.822840 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-h65vg"] Nov 03 10:28:43.826881 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:43.826852 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb00dfc2_12f0_4fc3_ba37_0bc83b344602.slice/crio-a22ba6a3a7548901cd56e023bbfc63e1b48b072eda04c0d8df49f3e880032de2 WatchSource:0}: Error finding container a22ba6a3a7548901cd56e023bbfc63e1b48b072eda04c0d8df49f3e880032de2: Status 404 returned error can't find the container with id a22ba6a3a7548901cd56e023bbfc63e1b48b072eda04c0d8df49f3e880032de2 Nov 03 10:28:43.828203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.828180 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ztmr2"] Nov 03 10:28:43.834471 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:43.834393 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb92e08_27a4_458a_818b_0215acd6dbd5.slice/crio-44f26c54bc828be08793f9a407da781a6605aa4e89c978b368ba802dee801aee WatchSource:0}: Error finding container 44f26c54bc828be08793f9a407da781a6605aa4e89c978b368ba802dee801aee: Status 404 returned error can't find the container with id 44f26c54bc828be08793f9a407da781a6605aa4e89c978b368ba802dee801aee Nov 03 10:28:43.871991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.871965 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:43.872099 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.872040 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:43.872196 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.872119 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:43.872196 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.872154 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:44.872130191 +0000 UTC m=+2.857001471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : configmap references non-existent config key: service-ca.crt Nov 03 10:28:43.872367 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.872214 5086 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 10:28:43.872367 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:43.872260 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:44.872245436 +0000 UTC m=+2.857116713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : secret "insights-runtime-extractor-tls" not found Nov 03 10:28:43.874271 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:43.874245 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea16e9-1545-4881-b7c2-883d8b09a6fe-metrics-tls\") pod \"dns-default-fb26v\" (UID: \"f8ea16e9-1545-4881-b7c2-883d8b09a6fe\") " pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:44.046569 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.046531 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:44.226174 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.226084 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" event={"ID":"7e537c04-9b2a-4d85-9ffd-db08c554c838","Type":"ContainerStarted","Data":"de9fd499935d97c765d882dbc2838b42004de777a137a3ef1281a6a78978ac31"} Nov 03 10:28:44.227255 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.227226 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ztmr2" event={"ID":"dcb92e08-27a4-458a-818b-0215acd6dbd5","Type":"ContainerStarted","Data":"44f26c54bc828be08793f9a407da781a6605aa4e89c978b368ba802dee801aee"} Nov 03 10:28:44.228422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.228392 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" event={"ID":"cb00dfc2-12f0-4fc3-ba37-0bc83b344602","Type":"ContainerStarted","Data":"a22ba6a3a7548901cd56e023bbfc63e1b48b072eda04c0d8df49f3e880032de2"} Nov 03 10:28:44.229790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.229768 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" event={"ID":"369e7292-b645-4eb6-9a5d-62e08910f1ef","Type":"ContainerStarted","Data":"6ec510ef48cec57b5d3572579781a9600c7461a485ccb7a258018d72e7b2069e"} Nov 03 10:28:44.229868 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.229805 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" event={"ID":"369e7292-b645-4eb6-9a5d-62e08910f1ef","Type":"ContainerStarted","Data":"cd25a392dece585f0a2ecc53e003bc66a2ca00573c4909d42c45e6c42df883a2"} Nov 03 10:28:44.231248 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.231229 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" event={"ID":"c634e6ef-e187-4409-a0a0-bae5793bc137","Type":"ContainerStarted","Data":"1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911"} Nov 03 10:28:44.231331 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.231261 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" event={"ID":"c634e6ef-e187-4409-a0a0-bae5793bc137","Type":"ContainerStarted","Data":"764c92b12e06a5383f9f6f0592405d4d510085cffd34fbe5ad2b4e3d0d723445"} Nov 03 10:28:44.233060 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.233024 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:28:44.233171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.233076 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" event={"ID":"aedffb88-1606-439b-968d-ab12c5a2b984","Type":"ContainerStarted","Data":"8b1efe49efad9c4b51691d258f952a30bdfaaeedb15148caefab01358a278c46"} Nov 03 10:28:44.337795 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.337651 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fb26v"] Nov 03 10:28:44.340869 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:44.340832 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ea16e9_1545_4881_b7c2_883d8b09a6fe.slice/crio-ba5c55a30f76f77927c481763272eded323fc41544f94a22036c33ac103d0ee8 WatchSource:0}: Error finding container ba5c55a30f76f77927c481763272eded323fc41544f94a22036c33ac103d0ee8: Status 404 returned error can't find the container with id ba5c55a30f76f77927c481763272eded323fc41544f94a22036c33ac103d0ee8 Nov 03 10:28:44.787227 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.787188 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:44.787490 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:44.787383 5086 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:44.787490 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:44.787464 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls podName:5a9404d9-1240-451f-8064-255583bb5da6 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:46.78744338 +0000 UTC m=+4.772314648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rchbs" (UID: "5a9404d9-1240-451f-8064-255583bb5da6") : secret "cluster-monitoring-operator-tls" not found Nov 03 10:28:44.887799 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.887763 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:44.887968 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:44.887831 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:44.888046 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:44.887980 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:46.887961889 +0000 UTC m=+4.872833169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : configmap references non-existent config key: service-ca.crt Nov 03 10:28:44.888490 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:44.888406 5086 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 10:28:44.888490 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:44.888459 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:46.888443849 +0000 UTC m=+4.873315125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : secret "insights-runtime-extractor-tls" not found Nov 03 10:28:45.240863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:45.240816 5086 generic.go:356] "Generic (PLEG): container finished" podID="5f7b33fa-69cf-4582-a4b4-844ed91e2e26" containerID="f8e3943ecc06063845be42fd2416be9fc173fd1c21ff4e900fd4befc89b64f6e" exitCode=0 Nov 03 10:28:45.241457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:45.240982 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8rp5q" event={"ID":"5f7b33fa-69cf-4582-a4b4-844ed91e2e26","Type":"ContainerDied","Data":"f8e3943ecc06063845be42fd2416be9fc173fd1c21ff4e900fd4befc89b64f6e"} Nov 03 10:28:45.241457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:45.241378 5086 scope.go:122] "RemoveContainer" containerID="f8e3943ecc06063845be42fd2416be9fc173fd1c21ff4e900fd4befc89b64f6e" Nov 03 10:28:45.243315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:45.243260 5086 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 10:28:45.244346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:45.243777 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fb26v" event={"ID":"f8ea16e9-1545-4881-b7c2-883d8b09a6fe","Type":"ContainerStarted","Data":"ba5c55a30f76f77927c481763272eded323fc41544f94a22036c33ac103d0ee8"} Nov 03 10:28:45.244346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:45.243852 5086 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 10:28:45.477056 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:45.475739 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:45.633130 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:45.633083 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:28:46.250425 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.250334 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ztmr2" event={"ID":"dcb92e08-27a4-458a-818b-0215acd6dbd5","Type":"ContainerStarted","Data":"0c6a90d8bda82f279c4049b80aef708fffe2bb2071d74c1048f3efa166832ec4"} Nov 03 10:28:46.250425 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.250404 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ztmr2" event={"ID":"dcb92e08-27a4-458a-818b-0215acd6dbd5","Type":"ContainerStarted","Data":"f325d2cf2c5c9b6e4b30b58d406babe9403c13391513e75086a51dab5b7bc5c9"} Nov 03 10:28:46.253061 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.253032 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-h65vg" event={"ID":"cb00dfc2-12f0-4fc3-ba37-0bc83b344602","Type":"ContainerStarted","Data":"0f302fe634d8a1a8e6f20aba7aab83da462ec246748381152acafa1083dec0fc"} Nov 03 10:28:46.255111 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.254984 5086 generic.go:356] "Generic (PLEG): container finished" podID="e8bf626d-af0f-47d8-8d61-686c390139af" containerID="bcaa8fc22e86b76bd9ae13f53da0bb7496202c5408d18746a33f84390d8f49da" exitCode=0 Nov 03 10:28:46.255111 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.255021 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" event={"ID":"e8bf626d-af0f-47d8-8d61-686c390139af","Type":"ContainerDied","Data":"bcaa8fc22e86b76bd9ae13f53da0bb7496202c5408d18746a33f84390d8f49da"} Nov 03 10:28:46.255387 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.255366 5086 scope.go:122] "RemoveContainer" containerID="bcaa8fc22e86b76bd9ae13f53da0bb7496202c5408d18746a33f84390d8f49da" Nov 03 10:28:46.258026 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.258001 5086 generic.go:356] "Generic (PLEG): container finished" podID="29923b63-cb6e-4a44-b098-ebd0317c1c14" containerID="95dccbf46d1b84b722664be252773094df825cfd4c39adbc3273686bea37f2d9" exitCode=0 Nov 03 10:28:46.258115 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.258077 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" event={"ID":"29923b63-cb6e-4a44-b098-ebd0317c1c14","Type":"ContainerDied","Data":"95dccbf46d1b84b722664be252773094df825cfd4c39adbc3273686bea37f2d9"} Nov 03 10:28:46.258669 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.258377 5086 scope.go:122] "RemoveContainer" containerID="95dccbf46d1b84b722664be252773094df825cfd4c39adbc3273686bea37f2d9" Nov 03 10:28:46.261102 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.261086 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8rp5q" event={"ID":"5f7b33fa-69cf-4582-a4b4-844ed91e2e26","Type":"ContainerStarted","Data":"fd9c6fc5c1e2fc4df85f63f6e3946ca8a7b3152f9169131a5ea7e454db12f2f9"} Nov 03 10:28:46.817977 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.817370 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:46.820675 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.820641 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a9404d9-1240-451f-8064-255583bb5da6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rchbs\" (UID: \"5a9404d9-1240-451f-8064-255583bb5da6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:46.918875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.918832 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:46.919083 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:46.918923 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:46.919083 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:46.919011 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle podName:0dbdd6e7-a8de-4990-9aec-358480af8521 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:50.918995166 +0000 UTC m=+8.903866428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle") pod "router-default-788447d7cc-tl5jd" (UID: "0dbdd6e7-a8de-4990-9aec-358480af8521") : configmap references non-existent config key: service-ca.crt Nov 03 10:28:46.919083 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:46.919022 5086 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 10:28:46.919083 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:28:46.919055 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls podName:0945eba8-6898-4dfe-ae35-687496c3b596 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:50.919049307 +0000 UTC m=+8.903920567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhvqw" (UID: "0945eba8-6898-4dfe-ae35-687496c3b596") : secret "insights-runtime-extractor-tls" not found Nov 03 10:28:47.028148 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:47.028111 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" Nov 03 10:28:47.270627 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:47.270591 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t4kgk" event={"ID":"e8bf626d-af0f-47d8-8d61-686c390139af","Type":"ContainerStarted","Data":"603f04bd1f72b32732f1a06386daeb15106277bbd3f2ec92a92ff8bc728f75e5"} Nov 03 10:28:47.272620 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:47.272591 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fb26v" event={"ID":"f8ea16e9-1545-4881-b7c2-883d8b09a6fe","Type":"ContainerStarted","Data":"3a3f1fd2cda73c98e801a64f3ec1d99dc706051cb3da23535425eb712c58fab7"} Nov 03 10:28:47.272781 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:47.272639 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fb26v" event={"ID":"f8ea16e9-1545-4881-b7c2-883d8b09a6fe","Type":"ContainerStarted","Data":"4800c15ed633cb87da981b0e21db684eb417fad73118045b33eb8b9d2ab6cefb"} Nov 03 10:28:47.272781 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:47.272715 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:47.274658 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:47.274632 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hhq6f" event={"ID":"29923b63-cb6e-4a44-b098-ebd0317c1c14","Type":"ContainerStarted","Data":"6c32e5aaa88fb41daca25fa05d401c27391403d5eb3c9a3aafe255f3b498692e"} Nov 03 10:28:47.276421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:47.276394 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" event={"ID":"7e537c04-9b2a-4d85-9ffd-db08c554c838","Type":"ContainerStarted","Data":"99ecd63eb52b40beab9295a4fae7cd833e10c22a0c4fabc1e83f2634553f792a"} Nov 03 10:28:47.276537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:47.276442 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69g9h" event={"ID":"7e537c04-9b2a-4d85-9ffd-db08c554c838","Type":"ContainerStarted","Data":"8155ddcfacf162e9b2429ecff2441ce2f23bb081056007f5eacb94789761e476"} Nov 03 10:28:47.299206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:47.299175 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs"] Nov 03 10:28:47.302427 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:47.302402 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9404d9_1240_451f_8064_255583bb5da6.slice/crio-233a4722cf39f871f2d5c5847965320891bf538a5b5d3eb15878c35f84c07284 WatchSource:0}: Error finding container 233a4722cf39f871f2d5c5847965320891bf538a5b5d3eb15878c35f84c07284: Status 404 returned error can't find the container with id 233a4722cf39f871f2d5c5847965320891bf538a5b5d3eb15878c35f84c07284 Nov 03 10:28:48.282835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:48.282788 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" event={"ID":"5a9404d9-1240-451f-8064-255583bb5da6","Type":"ContainerStarted","Data":"233a4722cf39f871f2d5c5847965320891bf538a5b5d3eb15878c35f84c07284"} Nov 03 10:28:48.317837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:48.317808 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:48.318022 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:48.318006 5086 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 10:28:48.518138 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:48.517448 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:48.518138 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:48.517598 5086 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 10:28:48.521914 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:48.521887 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2m2xr" Nov 03 10:28:50.294606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:50.294570 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rchbs" event={"ID":"5a9404d9-1240-451f-8064-255583bb5da6","Type":"ContainerStarted","Data":"04ec022fa10a8abb50674349322f93a617c21e8368f347b3a61484c198572602"} Nov 03 10:28:50.429513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:50.429478 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:28:50.695490 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:50.695448 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:28:50.955245 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:50.955172 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:50.955245 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:50.955239 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:50.955811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:50.955780 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbdd6e7-a8de-4990-9aec-358480af8521-service-ca-bundle\") pod \"router-default-788447d7cc-tl5jd\" (UID: \"0dbdd6e7-a8de-4990-9aec-358480af8521\") " pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:50.957573 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:50.957551 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0945eba8-6898-4dfe-ae35-687496c3b596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhvqw\" (UID: \"0945eba8-6898-4dfe-ae35-687496c3b596\") " pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:51.235939 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:51.235856 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:51.253933 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:51.253903 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhvqw" Nov 03 10:28:51.507673 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:51.507591 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-788447d7cc-tl5jd"] Nov 03 10:28:51.510758 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:51.510734 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbdd6e7_a8de_4990_9aec_358480af8521.slice/crio-926e0453e4f1feb0648941a9564811e6c1ecc384bc072efaa2097048391b22ef WatchSource:0}: Error finding container 926e0453e4f1feb0648941a9564811e6c1ecc384bc072efaa2097048391b22ef: Status 404 returned error can't find the container with id 926e0453e4f1feb0648941a9564811e6c1ecc384bc072efaa2097048391b22ef Nov 03 10:28:51.531749 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:51.531720 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rhvqw"] Nov 03 10:28:51.543503 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:51.543474 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0945eba8_6898_4dfe_ae35_687496c3b596.slice/crio-98d4f966bbd90eea0fa706202eb725a0cb2e71e15f2b9ae15e7aba9185de62d2 WatchSource:0}: Error finding container 98d4f966bbd90eea0fa706202eb725a0cb2e71e15f2b9ae15e7aba9185de62d2: Status 404 returned error can't find the container with id 98d4f966bbd90eea0fa706202eb725a0cb2e71e15f2b9ae15e7aba9185de62d2 Nov 03 10:28:52.305154 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:52.305125 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-788447d7cc-tl5jd" event={"ID":"0dbdd6e7-a8de-4990-9aec-358480af8521","Type":"ContainerStarted","Data":"368d73dedf172954dafe42369313c250a4086f4663ad1351e177e798ebbea082"} Nov 03 10:28:52.305307 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:52.305185 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-788447d7cc-tl5jd" event={"ID":"0dbdd6e7-a8de-4990-9aec-358480af8521","Type":"ContainerStarted","Data":"926e0453e4f1feb0648941a9564811e6c1ecc384bc072efaa2097048391b22ef"} Nov 03 10:28:52.307027 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:52.306997 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhvqw" event={"ID":"0945eba8-6898-4dfe-ae35-687496c3b596","Type":"ContainerStarted","Data":"ad76dbff74a905ae00800a0ad7292f50c069c3c51961d34c26ac57c659d5134c"} Nov 03 10:28:52.307108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:52.307064 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhvqw" event={"ID":"0945eba8-6898-4dfe-ae35-687496c3b596","Type":"ContainerStarted","Data":"98d4f966bbd90eea0fa706202eb725a0cb2e71e15f2b9ae15e7aba9185de62d2"} Nov 03 10:28:53.236382 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:53.236342 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:53.239379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:53.239350 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:53.314747 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:53.314704 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhvqw" event={"ID":"0945eba8-6898-4dfe-ae35-687496c3b596","Type":"ContainerStarted","Data":"7f98b9caafdbe5b62787c6f078727e5609f37e47b53299586eccd05da6d9cf90"} Nov 03 10:28:53.315013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:53.314994 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:53.316284 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:53.316262 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-788447d7cc-tl5jd" Nov 03 10:28:54.209790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:54.209761 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq8sx" Nov 03 10:28:54.321103 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:54.321066 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhvqw" event={"ID":"0945eba8-6898-4dfe-ae35-687496c3b596","Type":"ContainerStarted","Data":"18c9371bb59dfd6362687bda0b852b1fab00bcb1b2eeda9097e5bc3c50c92d6c"} Nov 03 10:28:57.117651 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.117603 5086 patch_prober.go:29] interesting pod/image-registry-678cf4bb67-v4rz9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 10:28:57.118129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.117708 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" podUID="369e7292-b645-4eb6-9a5d-62e08910f1ef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 10:28:57.285515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.285484 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fb26v" Nov 03 10:28:57.657575 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.657537 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-f74wv-nz4fj"] Nov 03 10:28:57.662970 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.662941 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.678218 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.678033 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f74wv-nz4fj"] Nov 03 10:28:57.687129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.682588 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn"] Nov 03 10:28:57.688839 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.688811 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" Nov 03 10:28:57.693073 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.693042 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5898f97979-m2vxp"] Nov 03 10:28:57.699839 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.699816 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-7plpl"] Nov 03 10:28:57.699969 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.699927 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.701553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.701533 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v5g2s\"" Nov 03 10:28:57.701794 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.701770 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 10:28:57.705330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.705247 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn"] Nov 03 10:28:57.705537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.705521 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675dcb4c7d-4p5mj"] Nov 03 10:28:57.706207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.706188 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:28:57.706924 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.706903 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5898f97979-m2vxp"] Nov 03 10:28:57.711790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.711772 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 10:28:57.716247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.711887 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 10:28:57.716247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.711971 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 10:28:57.716247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.712087 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 10:28:57.716247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.712159 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 10:28:57.716247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.712907 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7plpl"] Nov 03 10:28:57.765478 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.765445 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65b85b84d7-v8cxp"] Nov 03 10:28:57.769125 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.769099 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:57.783013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.782979 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65b85b84d7-v8cxp"] Nov 03 10:28:57.801088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.801056 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 10:28:57.801088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.801055 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qpnvs\"" Nov 03 10:28:57.801281 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.801143 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-4mgk4\"" Nov 03 10:28:57.801281 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.801249 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 10:28:57.826158 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826127 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hkn\" (UniqueName: \"kubernetes.io/projected/44bd20f9-05bf-4ca7-9da3-4fd87c311c2c-kube-api-access-86hkn\") pod \"downloads-6dd789c8b7-7plpl\" (UID: \"44bd20f9-05bf-4ca7-9da3-4fd87c311c2c\") " pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:28:57.826360 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826165 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qhv6\" (UniqueName: \"kubernetes.io/projected/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-kube-api-access-9qhv6\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.826360 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826183 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44bd20f9-05bf-4ca7-9da3-4fd87c311c2c-tmp\") pod \"downloads-6dd789c8b7-7plpl\" (UID: \"44bd20f9-05bf-4ca7-9da3-4fd87c311c2c\") " pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:28:57.826360 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826204 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-trusted-ca-bundle\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.826360 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826339 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cb353b40-b1b4-43cf-b7a2-9d1fa9d3f225-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qp2wn\" (UID: \"cb353b40-b1b4-43cf-b7a2-9d1fa9d3f225\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" Nov 03 10:28:57.826530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826370 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5a3a09a-d7cf-4f67-9b22-99456a86976b-archives-path\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.826530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826396 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-service-ca\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.826530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826415 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-serving-cert\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.826530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826460 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-oauth-serving-cert\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.826530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826517 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-oauth-config\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.826716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826551 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86kv\" (UniqueName: \"kubernetes.io/projected/e5a3a09a-d7cf-4f67-9b22-99456a86976b-kube-api-access-p86kv\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.826716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826583 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-tmp\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.826716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826610 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-config\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.826716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.826703 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-service-ca-bundle\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.927740 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.927644 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-trusted-ca\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:57.927740 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.927703 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hkn\" (UniqueName: \"kubernetes.io/projected/44bd20f9-05bf-4ca7-9da3-4fd87c311c2c-kube-api-access-86hkn\") pod \"downloads-6dd789c8b7-7plpl\" (UID: \"44bd20f9-05bf-4ca7-9da3-4fd87c311c2c\") " pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:28:57.927952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.927774 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qhv6\" (UniqueName: \"kubernetes.io/projected/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-kube-api-access-9qhv6\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.927952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.927803 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44bd20f9-05bf-4ca7-9da3-4fd87c311c2c-tmp\") pod \"downloads-6dd789c8b7-7plpl\" (UID: \"44bd20f9-05bf-4ca7-9da3-4fd87c311c2c\") " pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:28:57.927952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.927829 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-registry-certificates\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:57.927952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.927854 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbzr\" (UniqueName: \"kubernetes.io/projected/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-kube-api-access-xkbzr\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:57.927952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.927882 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-trusted-ca-bundle\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.928128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.927997 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cb353b40-b1b4-43cf-b7a2-9d1fa9d3f225-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qp2wn\" (UID: \"cb353b40-b1b4-43cf-b7a2-9d1fa9d3f225\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" Nov 03 10:28:57.928128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928099 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5a3a09a-d7cf-4f67-9b22-99456a86976b-archives-path\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.928220 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928129 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-registry-tls\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:57.928220 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928165 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-service-ca\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.928220 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928195 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-serving-cert\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.928403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928254 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-oauth-serving-cert\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.928403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928270 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44bd20f9-05bf-4ca7-9da3-4fd87c311c2c-tmp\") pod \"downloads-6dd789c8b7-7plpl\" (UID: \"44bd20f9-05bf-4ca7-9da3-4fd87c311c2c\") " pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:28:57.928403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928373 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-oauth-config\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.928629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928411 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-ca-trust-extracted\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:57.928629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928444 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86kv\" (UniqueName: \"kubernetes.io/projected/e5a3a09a-d7cf-4f67-9b22-99456a86976b-kube-api-access-p86kv\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.928629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928488 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-installation-pull-secrets\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:57.928629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928499 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5a3a09a-d7cf-4f67-9b22-99456a86976b-archives-path\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.928629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928534 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-tmp\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.928629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928577 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-image-registry-private-configuration\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:57.928629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928607 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-config\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.928964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928638 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-service-ca-bundle\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.928964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928694 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-bound-sa-token\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:57.929065 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.928966 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-service-ca\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.929065 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.929034 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-trusted-ca-bundle\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.929158 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.929092 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-tmp\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.929342 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.929323 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-service-ca-bundle\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.929378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.929365 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-config\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.929588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.929570 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-oauth-serving-cert\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.931135 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.931116 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-oauth-config\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.931135 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.931125 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cb353b40-b1b4-43cf-b7a2-9d1fa9d3f225-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qp2wn\" (UID: \"cb353b40-b1b4-43cf-b7a2-9d1fa9d3f225\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" Nov 03 10:28:57.931346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.931329 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-serving-cert\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.939052 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.939025 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qhv6\" (UniqueName: \"kubernetes.io/projected/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-kube-api-access-9qhv6\") pod \"console-5898f97979-m2vxp\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:57.939571 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.939544 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86kv\" (UniqueName: \"kubernetes.io/projected/e5a3a09a-d7cf-4f67-9b22-99456a86976b-kube-api-access-p86kv\") pod \"periodic-gathering-f74wv-nz4fj\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:57.939755 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.939732 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hkn\" (UniqueName: \"kubernetes.io/projected/44bd20f9-05bf-4ca7-9da3-4fd87c311c2c-kube-api-access-86hkn\") pod \"downloads-6dd789c8b7-7plpl\" (UID: \"44bd20f9-05bf-4ca7-9da3-4fd87c311c2c\") " pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:28:57.986344 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:57.986288 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:28:58.005368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.005337 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" Nov 03 10:28:58.021617 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.021438 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:28:58.028892 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.028690 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:28:58.029356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.029328 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-registry-certificates\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.029462 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.029367 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbzr\" (UniqueName: \"kubernetes.io/projected/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-kube-api-access-xkbzr\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.029462 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.029406 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-registry-tls\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.029462 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.029436 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-ca-trust-extracted\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.029462 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.029459 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-installation-pull-secrets\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.029728 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.029492 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-image-registry-private-configuration\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.029728 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.029516 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-bound-sa-token\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.029728 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.029533 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-trusted-ca\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.030054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.030028 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-ca-trust-extracted\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.030828 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.030803 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-registry-certificates\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.031953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.031927 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-trusted-ca\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.033169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.033148 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-image-registry-private-configuration\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.033418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.033396 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-installation-pull-secrets\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.035573 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.035530 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-registry-tls\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.043610 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.043260 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-bound-sa-token\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.046752 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.046455 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbzr\" (UniqueName: \"kubernetes.io/projected/e00ff8a7-26cf-4089-a4da-7ddb27de2a75-kube-api-access-xkbzr\") pod \"image-registry-65b85b84d7-v8cxp\" (UID: \"e00ff8a7-26cf-4089-a4da-7ddb27de2a75\") " pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.081032 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.080994 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:58.266719 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.266572 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f74wv-nz4fj"] Nov 03 10:28:58.269015 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:58.268982 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a3a09a_d7cf_4f67_9b22_99456a86976b.slice/crio-05a25239a2f065f6dbf1bd287a0c32726c5f5bce55859ede9e87b62248ea0574 WatchSource:0}: Error finding container 05a25239a2f065f6dbf1bd287a0c32726c5f5bce55859ede9e87b62248ea0574: Status 404 returned error can't find the container with id 05a25239a2f065f6dbf1bd287a0c32726c5f5bce55859ede9e87b62248ea0574 Nov 03 10:28:58.298720 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.298663 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn"] Nov 03 10:28:58.302557 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:58.302526 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb353b40_b1b4_43cf_b7a2_9d1fa9d3f225.slice/crio-04b320485e2001c3750eb0740f884b88c86c0fcfc9deb2cd00767eaa7ffce4ff WatchSource:0}: Error finding container 04b320485e2001c3750eb0740f884b88c86c0fcfc9deb2cd00767eaa7ffce4ff: Status 404 returned error can't find the container with id 04b320485e2001c3750eb0740f884b88c86c0fcfc9deb2cd00767eaa7ffce4ff Nov 03 10:28:58.325839 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.325798 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7plpl"] Nov 03 10:28:58.328683 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:58.328478 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bd20f9_05bf_4ca7_9da3_4fd87c311c2c.slice/crio-e76dc0a4458ba250cd088ba873d5fcd8823218baf238f496f67265fe95c1d6aa WatchSource:0}: Error finding container e76dc0a4458ba250cd088ba873d5fcd8823218baf238f496f67265fe95c1d6aa: Status 404 returned error can't find the container with id e76dc0a4458ba250cd088ba873d5fcd8823218baf238f496f67265fe95c1d6aa Nov 03 10:28:58.333673 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.333646 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5898f97979-m2vxp"] Nov 03 10:28:58.338672 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:58.338057 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0bf7e5_e4e5_4aa3_98b1_d82b82ccceed.slice/crio-31525cc6eceffd531cfef13362a5a9b302c6978c30064ea74cdfc433c4a2c7c4 WatchSource:0}: Error finding container 31525cc6eceffd531cfef13362a5a9b302c6978c30064ea74cdfc433c4a2c7c4: Status 404 returned error can't find the container with id 31525cc6eceffd531cfef13362a5a9b302c6978c30064ea74cdfc433c4a2c7c4 Nov 03 10:28:58.344666 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.344640 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" event={"ID":"cb353b40-b1b4-43cf-b7a2-9d1fa9d3f225","Type":"ContainerStarted","Data":"04b320485e2001c3750eb0740f884b88c86c0fcfc9deb2cd00767eaa7ffce4ff"} Nov 03 10:28:58.345882 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.345856 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7plpl" event={"ID":"44bd20f9-05bf-4ca7-9da3-4fd87c311c2c","Type":"ContainerStarted","Data":"e76dc0a4458ba250cd088ba873d5fcd8823218baf238f496f67265fe95c1d6aa"} Nov 03 10:28:58.347246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.347224 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f74wv-nz4fj" event={"ID":"e5a3a09a-d7cf-4f67-9b22-99456a86976b","Type":"ContainerStarted","Data":"05a25239a2f065f6dbf1bd287a0c32726c5f5bce55859ede9e87b62248ea0574"} Nov 03 10:28:58.397338 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:58.397285 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65b85b84d7-v8cxp"] Nov 03 10:28:58.400311 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:28:58.400263 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00ff8a7_26cf_4089_a4da_7ddb27de2a75.slice/crio-85d0fb0b9822fecabc3df4ee524b4c0f7fb585fe669175186222bdcd969956bc WatchSource:0}: Error finding container 85d0fb0b9822fecabc3df4ee524b4c0f7fb585fe669175186222bdcd969956bc: Status 404 returned error can't find the container with id 85d0fb0b9822fecabc3df4ee524b4c0f7fb585fe669175186222bdcd969956bc Nov 03 10:28:59.357551 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:59.356635 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f74wv-nz4fj" event={"ID":"e5a3a09a-d7cf-4f67-9b22-99456a86976b","Type":"ContainerStarted","Data":"16a8a1ff97f2227aa5f367c3147b4a4b3b200368a38ebb54500baf507b78e855"} Nov 03 10:28:59.361028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:59.360920 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" event={"ID":"e00ff8a7-26cf-4089-a4da-7ddb27de2a75","Type":"ContainerStarted","Data":"20729824128a678c20a0f2a2cdcc6b7d3fcfa85b0ae3c636bf689783086ecd36"} Nov 03 10:28:59.361028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:59.360963 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" event={"ID":"e00ff8a7-26cf-4089-a4da-7ddb27de2a75","Type":"ContainerStarted","Data":"85d0fb0b9822fecabc3df4ee524b4c0f7fb585fe669175186222bdcd969956bc"} Nov 03 10:28:59.361028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:59.360986 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:28:59.362763 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:59.362732 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5898f97979-m2vxp" event={"ID":"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed","Type":"ContainerStarted","Data":"31525cc6eceffd531cfef13362a5a9b302c6978c30064ea74cdfc433c4a2c7c4"} Nov 03 10:28:59.372713 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:28:59.372614 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-f74wv-nz4fj" podStartSLOduration=2.372598185 podStartE2EDuration="2.372598185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:59.372101277 +0000 UTC m=+17.356972569" watchObservedRunningTime="2026-11-03 10:28:59.372598185 +0000 UTC m=+17.357469466" Nov 03 10:29:00.370192 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.370146 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" event={"ID":"cb353b40-b1b4-43cf-b7a2-9d1fa9d3f225","Type":"ContainerStarted","Data":"952ee557983bef5718f65552339d100bd3ebe99c80dfbbb00a005ed9fc550d5b"} Nov 03 10:29:00.370744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.370721 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" Nov 03 10:29:00.376045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.376021 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" Nov 03 10:29:00.385423 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.385362 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp2wn" podStartSLOduration=2.116806286 podStartE2EDuration="3.385343853s" totalImagesPullingTime="1.268537567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:00.384966412 +0000 UTC m=+18.369837698" watchObservedRunningTime="2026-11-03 10:29:00.385343853 +0000 UTC m=+18.370215134" Nov 03 10:29:00.385992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.385954 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" podStartSLOduration=3.385942178 podStartE2EDuration="3.385942178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:59.392039481 +0000 UTC m=+17.376910765" watchObservedRunningTime="2026-11-03 10:29:00.385942178 +0000 UTC m=+18.370813469" Nov 03 10:29:00.435149 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.435120 5086 patch_prober.go:29] interesting pod/image-registry-678cf4bb67-v4rz9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 10:29:00.435264 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.435171 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" podUID="369e7292-b645-4eb6-9a5d-62e08910f1ef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 10:29:00.800271 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.800174 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2"] Nov 03 10:29:00.805666 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.805640 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:00.813249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.813218 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 10:29:00.813429 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.813409 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lsprr\"" Nov 03 10:29:00.813603 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.813505 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 10:29:00.813603 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.813567 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 10:29:00.814050 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.813986 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2"] Nov 03 10:29:00.957806 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.957761 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4df3ca90-d4ac-4e97-9056-76645061f149-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:00.957963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.957864 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4df3ca90-d4ac-4e97-9056-76645061f149-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:00.957963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.957896 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9cc\" (UniqueName: \"kubernetes.io/projected/4df3ca90-d4ac-4e97-9056-76645061f149-kube-api-access-8r9cc\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:00.958053 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:00.957964 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4df3ca90-d4ac-4e97-9056-76645061f149-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.058621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.058528 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4df3ca90-d4ac-4e97-9056-76645061f149-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.058621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.058581 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9cc\" (UniqueName: \"kubernetes.io/projected/4df3ca90-d4ac-4e97-9056-76645061f149-kube-api-access-8r9cc\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.058860 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.058660 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4df3ca90-d4ac-4e97-9056-76645061f149-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.058860 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.058709 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4df3ca90-d4ac-4e97-9056-76645061f149-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.060172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.060138 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4df3ca90-d4ac-4e97-9056-76645061f149-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.061900 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.061872 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4df3ca90-d4ac-4e97-9056-76645061f149-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.062041 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.062015 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4df3ca90-d4ac-4e97-9056-76645061f149-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.067497 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.067470 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9cc\" (UniqueName: \"kubernetes.io/projected/4df3ca90-d4ac-4e97-9056-76645061f149-kube-api-access-8r9cc\") pod \"prometheus-operator-9cd89b74c-xmlk2\" (UID: \"4df3ca90-d4ac-4e97-9056-76645061f149\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.132565 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.132527 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" Nov 03 10:29:01.421589 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:01.421548 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2"] Nov 03 10:29:01.429474 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:29:01.429438 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df3ca90_d4ac_4e97_9056_76645061f149.slice/crio-c6d6a8014e834c72542c2fcb24f61da404e1f25841a79eac2b38fbc2bc49d7d2 WatchSource:0}: Error finding container c6d6a8014e834c72542c2fcb24f61da404e1f25841a79eac2b38fbc2bc49d7d2: Status 404 returned error can't find the container with id c6d6a8014e834c72542c2fcb24f61da404e1f25841a79eac2b38fbc2bc49d7d2 Nov 03 10:29:02.384279 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:02.384237 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" event={"ID":"4df3ca90-d4ac-4e97-9056-76645061f149","Type":"ContainerStarted","Data":"c6d6a8014e834c72542c2fcb24f61da404e1f25841a79eac2b38fbc2bc49d7d2"} Nov 03 10:29:02.398132 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:02.398095 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5898f97979-m2vxp" event={"ID":"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed","Type":"ContainerStarted","Data":"6e4aa70e38f4e4a2e2aed7430482096be2ddb97ac5c0b2a7f0e4687da3a50133"} Nov 03 10:29:02.416271 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:02.416210 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5898f97979-m2vxp" podStartSLOduration=1.828302011 podStartE2EDuration="5.416195592s" totalImagesPullingTime="3.587893581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:02.414531065 +0000 UTC m=+20.399402352" watchObservedRunningTime="2026-11-03 10:29:02.416195592 +0000 UTC m=+20.401066875" Nov 03 10:29:03.407727 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:03.407688 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" event={"ID":"4df3ca90-d4ac-4e97-9056-76645061f149","Type":"ContainerStarted","Data":"3e3cb3d7cd0f831768459b101653cee99dd004de389e74588432f3e20c4358f0"} Nov 03 10:29:03.408194 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:03.407787 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" event={"ID":"4df3ca90-d4ac-4e97-9056-76645061f149","Type":"ContainerStarted","Data":"ba631289f84a927c99bc65d9183fdd7d4b74430ce0035edfc73b1cb63101e197"} Nov 03 10:29:03.424581 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:03.424518 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xmlk2" podStartSLOduration=1.6414423729999998 podStartE2EDuration="3.42449776s" totalImagesPullingTime="1.783055387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:03.422322411 +0000 UTC m=+21.407193700" watchObservedRunningTime="2026-11-03 10:29:03.42449776 +0000 UTC m=+21.409369046" Nov 03 10:29:03.859535 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:03.859497 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-h65vg_cb00dfc2-12f0-4fc3-ba37-0bc83b344602/networking-console-plugin/0.log" Nov 03 10:29:04.041737 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:04.041663 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:29:04.244202 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:04.244113 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/2.log" Nov 03 10:29:04.453325 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:04.451496 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5898f97979-m2vxp_bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed/console/0.log" Nov 03 10:29:04.841694 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:04.841665 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fb26v_f8ea16e9-1545-4881-b7c2-883d8b09a6fe/dns/0.log" Nov 03 10:29:05.042374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.042344 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fb26v_f8ea16e9-1545-4881-b7c2-883d8b09a6fe/kube-rbac-proxy/0.log" Nov 03 10:29:05.192441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.191362 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wr9wd"] Nov 03 10:29:05.226557 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.226514 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.239995 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.239686 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjjb4\"" Nov 03 10:29:05.239995 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.239785 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 10:29:05.239995 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.239853 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 10:29:05.296610 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.296570 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.296802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.296634 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625af666-e1d5-4bd0-b747-84cbae4aee5a-metrics-client-ca\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.296802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.296727 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/625af666-e1d5-4bd0-b747-84cbae4aee5a-root\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.296802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.296767 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-tls\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.296802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.296788 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.297020 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.296948 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgj76\" (UniqueName: \"kubernetes.io/projected/625af666-e1d5-4bd0-b747-84cbae4aee5a-kube-api-access-wgj76\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.297071 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.297023 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-wtmp\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.297071 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.297047 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/625af666-e1d5-4bd0-b747-84cbae4aee5a-sys\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.297170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.297070 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-textfile\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.320901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.320863 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 10:29:05.397669 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397632 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.397831 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397697 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625af666-e1d5-4bd0-b747-84cbae4aee5a-metrics-client-ca\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.397831 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397730 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/625af666-e1d5-4bd0-b747-84cbae4aee5a-root\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.397831 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397752 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-tls\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.397831 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397777 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.397831 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397827 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgj76\" (UniqueName: \"kubernetes.io/projected/625af666-e1d5-4bd0-b747-84cbae4aee5a-kube-api-access-wgj76\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.398141 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397877 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-wtmp\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.398141 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397903 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/625af666-e1d5-4bd0-b747-84cbae4aee5a-sys\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.398141 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397928 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-textfile\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.398141 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.398039 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-wtmp\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.398141 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.397826 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/625af666-e1d5-4bd0-b747-84cbae4aee5a-root\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.398141 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.398120 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/625af666-e1d5-4bd0-b747-84cbae4aee5a-sys\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.398406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.398235 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-textfile\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.398406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.398279 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.398406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.398307 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625af666-e1d5-4bd0-b747-84cbae4aee5a-metrics-client-ca\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.400410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.400380 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.400575 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.400552 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/625af666-e1d5-4bd0-b747-84cbae4aee5a-node-exporter-tls\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.405577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.405555 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgj76\" (UniqueName: \"kubernetes.io/projected/625af666-e1d5-4bd0-b747-84cbae4aee5a-kube-api-access-wgj76\") pod \"node-exporter-wr9wd\" (UID: \"625af666-e1d5-4bd0-b747-84cbae4aee5a\") " pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.597949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.597907 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wr9wd" Nov 03 10:29:05.628128 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:29:05.628085 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625af666_e1d5_4bd0_b747_84cbae4aee5a.slice/crio-595d00e61d8812aecc707c12f85d034b3512d234e17035eca4d988cfa3ea170b WatchSource:0}: Error finding container 595d00e61d8812aecc707c12f85d034b3512d234e17035eca4d988cfa3ea170b: Status 404 returned error can't find the container with id 595d00e61d8812aecc707c12f85d034b3512d234e17035eca4d988cfa3ea170b Nov 03 10:29:05.643003 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.642976 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jhp98_35d6f6db-753a-4d1e-ae56-58659a5baab1/dns-node-resolver/0.log" Nov 03 10:29:05.843352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:05.843318 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65b85b84d7-v8cxp_e00ff8a7-26cf-4089-a4da-7ddb27de2a75/registry/0.log" Nov 03 10:29:06.042052 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:06.041978 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-675dcb4c7d-4p5mj_c634e6ef-e187-4409-a0a0-bae5793bc137/registry/0.log" Nov 03 10:29:06.242972 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:06.242935 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-678cf4bb67-v4rz9_369e7292-b645-4eb6-9a5d-62e08910f1ef/registry/0.log" Nov 03 10:29:06.431648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:06.431514 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wr9wd" event={"ID":"625af666-e1d5-4bd0-b747-84cbae4aee5a","Type":"ContainerStarted","Data":"595d00e61d8812aecc707c12f85d034b3512d234e17035eca4d988cfa3ea170b"} Nov 03 10:29:06.843152 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:06.843125 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ssnbk_35dabb24-0583-4de3-9a5d-21c8ef04c7c5/node-ca/0.log" Nov 03 10:29:07.042343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:07.042314 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-788447d7cc-tl5jd_0dbdd6e7-a8de-4990-9aec-358480af8521/router/0.log" Nov 03 10:29:07.243520 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:07.243493 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhr4s_4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8/serve-healthcheck-canary/0.log" Nov 03 10:29:07.439585 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:07.439543 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wr9wd" event={"ID":"625af666-e1d5-4bd0-b747-84cbae4aee5a","Type":"ContainerStarted","Data":"81cac294172faa6a4349c7778c0bbf5e0e5914492c25946cafc367f953462900"} Nov 03 10:29:07.713993 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:07.713958 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:29:08.022464 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:08.022383 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:29:08.022921 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:08.022522 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:29:08.028001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:08.027973 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:29:08.446564 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:08.446524 5086 generic.go:356] "Generic (PLEG): container finished" podID="625af666-e1d5-4bd0-b747-84cbae4aee5a" containerID="81cac294172faa6a4349c7778c0bbf5e0e5914492c25946cafc367f953462900" exitCode=0 Nov 03 10:29:08.446773 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:08.446591 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wr9wd" event={"ID":"625af666-e1d5-4bd0-b747-84cbae4aee5a","Type":"ContainerDied","Data":"81cac294172faa6a4349c7778c0bbf5e0e5914492c25946cafc367f953462900"} Nov 03 10:29:08.451720 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:08.451697 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:29:09.453885 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:09.453849 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wr9wd" event={"ID":"625af666-e1d5-4bd0-b747-84cbae4aee5a","Type":"ContainerStarted","Data":"ba3be769e6d9c23bd8c179dc47bffb41c6098f3422ea673faa864e3b423b2786"} Nov 03 10:29:09.454372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:09.453924 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wr9wd" event={"ID":"625af666-e1d5-4bd0-b747-84cbae4aee5a","Type":"ContainerStarted","Data":"0e4f992b61ac5b0801224c7834fdd4d36596df3e6a0f04697aa74f3f0ed85d8f"} Nov 03 10:29:09.472984 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:09.472931 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wr9wd" podStartSLOduration=2.792615108 podStartE2EDuration="4.472917361s" totalImagesPullingTime="1.587719607s" totalInitContainerRuntime="92.582646ms" isStatefulPod=true podCreationTimestamp="2026-11-03 10:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:09.471739614 +0000 UTC m=+27.456610923" watchObservedRunningTime="2026-11-03 10:29:09.472917361 +0000 UTC m=+27.457788698" Nov 03 10:29:09.528484 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:09.528442 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x9pz8" Nov 03 10:29:10.434469 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:10.434431 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:29:15.636719 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:15.636686 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t8sqq" Nov 03 10:29:17.390613 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:17.390570 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5898f97979-m2vxp"] Nov 03 10:29:17.506833 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:17.506791 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7plpl" event={"ID":"44bd20f9-05bf-4ca7-9da3-4fd87c311c2c","Type":"ContainerStarted","Data":"1636348c97d13dfdf011f4e7a93f3639d2253e8c8acb2d6f630abae7e7172e84"} Nov 03 10:29:17.507049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:17.506864 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:29:17.519185 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:17.519156 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-7plpl" Nov 03 10:29:17.524224 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:17.524166 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-7plpl" podStartSLOduration=2.358097094 podStartE2EDuration="20.524147193s" totalImagesPullingTime="18.166050099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:17.521700879 +0000 UTC m=+35.506572164" watchObservedRunningTime="2026-11-03 10:29:17.524147193 +0000 UTC m=+35.509018477" Nov 03 10:29:20.375643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:20.375613 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65b85b84d7-v8cxp" Nov 03 10:29:20.428456 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:20.428409 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678cf4bb67-v4rz9"] Nov 03 10:29:22.735755 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:22.735656 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" podUID="c634e6ef-e187-4409-a0a0-bae5793bc137" containerName="registry" containerID="cri-o://1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911" gracePeriod=30 Nov 03 10:29:23.025784 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.025756 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:29:23.099526 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.099483 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted\") pod \"c634e6ef-e187-4409-a0a0-bae5793bc137\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " Nov 03 10:29:23.099726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.099539 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca\") pod \"c634e6ef-e187-4409-a0a0-bae5793bc137\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " Nov 03 10:29:23.099726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.099571 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") pod \"c634e6ef-e187-4409-a0a0-bae5793bc137\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " Nov 03 10:29:23.099726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.099618 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c\") pod \"c634e6ef-e187-4409-a0a0-bae5793bc137\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " Nov 03 10:29:23.099726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.099700 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates\") pod \"c634e6ef-e187-4409-a0a0-bae5793bc137\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " Nov 03 10:29:23.099967 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.099752 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token\") pod \"c634e6ef-e187-4409-a0a0-bae5793bc137\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " Nov 03 10:29:23.099967 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.099784 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration\") pod \"c634e6ef-e187-4409-a0a0-bae5793bc137\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " Nov 03 10:29:23.099967 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.099836 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets\") pod \"c634e6ef-e187-4409-a0a0-bae5793bc137\" (UID: \"c634e6ef-e187-4409-a0a0-bae5793bc137\") " Nov 03 10:29:23.101615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.101573 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca" pod "c634e6ef-e187-4409-a0a0-bae5793bc137" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:29:23.102774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.101724 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates" pod "c634e6ef-e187-4409-a0a0-bae5793bc137" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:29:23.114583 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.112698 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted" pod "c634e6ef-e187-4409-a0a0-bae5793bc137" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:29:23.118470 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.118422 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration" pod "c634e6ef-e187-4409-a0a0-bae5793bc137" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:29:23.118637 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.118422 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c" pod "c634e6ef-e187-4409-a0a0-bae5793bc137" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137"). InnerVolumeSpecName "kube-api-access-8qk2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:29:23.118637 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.118428 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls" pod "c634e6ef-e187-4409-a0a0-bae5793bc137" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:29:23.118637 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.118557 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets" pod "c634e6ef-e187-4409-a0a0-bae5793bc137" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:29:23.118883 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.118818 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token" pod "c634e6ef-e187-4409-a0a0-bae5793bc137" (UID: "c634e6ef-e187-4409-a0a0-bae5793bc137"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:29:23.201252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.201209 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qk2c\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-kube-api-access-8qk2c\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:23.201252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.201250 5086 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-certificates\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:23.201555 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.201266 5086 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-bound-sa-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:23.201555 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.201278 5086 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-image-registry-private-configuration\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:23.201555 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.201288 5086 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c634e6ef-e187-4409-a0a0-bae5793bc137-installation-pull-secrets\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:23.201555 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.201315 5086 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c634e6ef-e187-4409-a0a0-bae5793bc137-ca-trust-extracted\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:23.201555 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.201325 5086 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c634e6ef-e187-4409-a0a0-bae5793bc137-trusted-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:23.201555 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.201334 5086 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c634e6ef-e187-4409-a0a0-bae5793bc137-registry-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:23.542890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.542849 5086 generic.go:356] "Generic (PLEG): container finished" podID="c634e6ef-e187-4409-a0a0-bae5793bc137" containerID="1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911" exitCode=0 Nov 03 10:29:23.543108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.542939 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" Nov 03 10:29:23.543108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.542937 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" event={"ID":"c634e6ef-e187-4409-a0a0-bae5793bc137","Type":"ContainerDied","Data":"1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911"} Nov 03 10:29:23.543108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.542997 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675dcb4c7d-4p5mj" event={"ID":"c634e6ef-e187-4409-a0a0-bae5793bc137","Type":"ContainerDied","Data":"764c92b12e06a5383f9f6f0592405d4d510085cffd34fbe5ad2b4e3d0d723445"} Nov 03 10:29:23.543108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.543029 5086 scope.go:122] "RemoveContainer" containerID="1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911" Nov 03 10:29:23.555011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.554992 5086 scope.go:122] "RemoveContainer" containerID="1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911" Nov 03 10:29:23.555397 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:29:23.555348 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911\": container with ID starting with 1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911 not found: ID does not exist" containerID="1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911" Nov 03 10:29:23.555502 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.555393 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911"} err="failed to get container status \"1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911\": rpc error: code = NotFound desc = could not find container \"1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911\": container with ID starting with 1a02cceb3b0c9f9d8979c5607c6ef831de6e89ab840cb1c101de88ddee463911 not found: ID does not exist" Nov 03 10:29:23.567361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.567334 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675dcb4c7d-4p5mj"] Nov 03 10:29:23.570766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:23.570741 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-675dcb4c7d-4p5mj"] Nov 03 10:29:24.170508 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:24.170467 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c634e6ef-e187-4409-a0a0-bae5793bc137" path="/var/lib/kubelet/pods/c634e6ef-e187-4409-a0a0-bae5793bc137/volumes" Nov 03 10:29:24.297874 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:29:24.297835 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:29:30.632290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:30.632258 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2j5wq-fjfdd"] Nov 03 10:29:30.632830 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:30.632522 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" podUID="0bbacc63-2772-4fa1-a868-afc6bc41fa44" containerName="insights-gathering" containerID="cri-o://6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba" gracePeriod=30 Nov 03 10:29:30.978522 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:30.978495 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:29:31.070777 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.070740 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path\") pod \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " Nov 03 10:29:31.070777 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.070784 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle\") pod \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " Nov 03 10:29:31.071033 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.070896 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle\") pod \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " Nov 03 10:29:31.071033 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.070927 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w\" (UniqueName: \"kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w\") pod \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\" (UID: \"0bbacc63-2772-4fa1-a868-afc6bc41fa44\") " Nov 03 10:29:31.071170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.071143 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path" pod "0bbacc63-2772-4fa1-a868-afc6bc41fa44" (UID: "0bbacc63-2772-4fa1-a868-afc6bc41fa44"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:29:31.071286 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.071263 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle" pod "0bbacc63-2772-4fa1-a868-afc6bc41fa44" (UID: "0bbacc63-2772-4fa1-a868-afc6bc41fa44"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:29:31.071371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.071259 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle" pod "0bbacc63-2772-4fa1-a868-afc6bc41fa44" (UID: "0bbacc63-2772-4fa1-a868-afc6bc41fa44"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:29:31.073534 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.073504 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w" pod "0bbacc63-2772-4fa1-a868-afc6bc41fa44" (UID: "0bbacc63-2772-4fa1-a868-afc6bc41fa44"). InnerVolumeSpecName "kube-api-access-5lz4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:29:31.171835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.171802 5086 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-service-ca-bundle\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:31.171835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.171831 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lz4w\" (UniqueName: \"kubernetes.io/projected/0bbacc63-2772-4fa1-a868-afc6bc41fa44-kube-api-access-5lz4w\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:31.172046 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.171846 5086 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0bbacc63-2772-4fa1-a868-afc6bc41fa44-archives-path\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:31.172046 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.171857 5086 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bbacc63-2772-4fa1-a868-afc6bc41fa44-trusted-ca-bundle\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:31.588536 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.588493 5086 generic.go:356] "Generic (PLEG): container finished" podID="0bbacc63-2772-4fa1-a868-afc6bc41fa44" containerID="6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba" exitCode=2 Nov 03 10:29:31.588717 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.588548 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" Nov 03 10:29:31.588717 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.588574 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" event={"ID":"0bbacc63-2772-4fa1-a868-afc6bc41fa44","Type":"ContainerDied","Data":"6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba"} Nov 03 10:29:31.588717 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.588632 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2j5wq-fjfdd" event={"ID":"0bbacc63-2772-4fa1-a868-afc6bc41fa44","Type":"ContainerDied","Data":"c4a2af27c2a8965d975bb507d67b8d7913a9892064d4c08b198143ada77b09ca"} Nov 03 10:29:31.588717 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.588653 5086 scope.go:122] "RemoveContainer" containerID="6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba" Nov 03 10:29:31.598059 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.598034 5086 scope.go:122] "RemoveContainer" containerID="6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba" Nov 03 10:29:31.598332 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:29:31.598308 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba\": container with ID starting with 6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba not found: ID does not exist" containerID="6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba" Nov 03 10:29:31.598402 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.598342 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba"} err="failed to get container status \"6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba\": rpc error: code = NotFound desc = could not find container \"6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba\": container with ID starting with 6bd426eba14ae34ddb7aad358d663c642266cc8fddf11bab7fc8c6dfc56ea4ba not found: ID does not exist" Nov 03 10:29:31.609311 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.609248 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2j5wq-fjfdd"] Nov 03 10:29:31.612876 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:31.612854 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-2j5wq-fjfdd"] Nov 03 10:29:32.169092 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:32.169049 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bbacc63-2772-4fa1-a868-afc6bc41fa44" path="/var/lib/kubelet/pods/0bbacc63-2772-4fa1-a868-afc6bc41fa44/volumes" Nov 03 10:29:42.422145 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.422039 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5898f97979-m2vxp" podUID="bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" containerName="console" containerID="cri-o://6e4aa70e38f4e4a2e2aed7430482096be2ddb97ac5c0b2a7f0e4687da3a50133" gracePeriod=15 Nov 03 10:29:42.639069 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.639034 5086 generic.go:356] "Generic (PLEG): container finished" podID="bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" containerID="6e4aa70e38f4e4a2e2aed7430482096be2ddb97ac5c0b2a7f0e4687da3a50133" exitCode=0 Nov 03 10:29:42.639249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.639077 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5898f97979-m2vxp" event={"ID":"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed","Type":"ContainerDied","Data":"6e4aa70e38f4e4a2e2aed7430482096be2ddb97ac5c0b2a7f0e4687da3a50133"} Nov 03 10:29:42.671288 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.671263 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:29:42.677826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.677770 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-oauth-config\") pod \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " Nov 03 10:29:42.677907 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.677841 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-kube-api-access-9qhv6\" (UniqueName: \"kubernetes.io/projected/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-kube-api-access-9qhv6\") pod \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " Nov 03 10:29:42.677907 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.677876 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-service-ca\") pod \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " Nov 03 10:29:42.677907 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.677904 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-config\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-config\") pod \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " Nov 03 10:29:42.678043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.677929 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-serving-cert\") pod \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " Nov 03 10:29:42.678043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.677958 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-oauth-serving-cert\") pod \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " Nov 03 10:29:42.678043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.678011 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-tmp\") pod \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\" (UID: \"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed\") " Nov 03 10:29:42.678409 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.678364 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-service-ca" pod "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" (UID: "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:29:42.678409 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.678390 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-config" pod "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" (UID: "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:29:42.678563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.678419 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-tmp" pod "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" (UID: "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:29:42.678563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.678454 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-oauth-serving-cert" pod "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" (UID: "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:29:42.680137 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.680116 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-kube-api-access-9qhv6" pod "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" (UID: "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed"). InnerVolumeSpecName "kube-api-access-9qhv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:29:42.680241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.680118 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-serving-cert" pod "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" (UID: "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:29:42.680241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.680145 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-oauth-config" pod "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" (UID: "bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:29:42.778725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.778681 5086 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:42.778725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.778716 5086 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-config\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:42.778725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.778727 5086 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-serving-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:42.778725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.778736 5086 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-oauth-serving-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:42.778989 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.778746 5086 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-tmp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:42.778989 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.778754 5086 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-console-oauth-config\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:42.778989 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:42.778764 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qhv6\" (UniqueName: \"kubernetes.io/projected/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed-kube-api-access-9qhv6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:43.644974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:43.644945 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5898f97979-m2vxp" Nov 03 10:29:43.644974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:43.644964 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5898f97979-m2vxp" event={"ID":"bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed","Type":"ContainerDied","Data":"31525cc6eceffd531cfef13362a5a9b302c6978c30064ea74cdfc433c4a2c7c4"} Nov 03 10:29:43.645485 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:43.645029 5086 scope.go:122] "RemoveContainer" containerID="6e4aa70e38f4e4a2e2aed7430482096be2ddb97ac5c0b2a7f0e4687da3a50133" Nov 03 10:29:43.665506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:43.665481 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5898f97979-m2vxp"] Nov 03 10:29:43.669245 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:43.669222 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5898f97979-m2vxp"] Nov 03 10:29:44.168938 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:44.168899 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" path="/var/lib/kubelet/pods/bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed/volumes" Nov 03 10:29:45.463400 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.463355 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" podUID="369e7292-b645-4eb6-9a5d-62e08910f1ef" containerName="registry" containerID="cri-o://6ec510ef48cec57b5d3572579781a9600c7461a485ccb7a258018d72e7b2069e" gracePeriod=30 Nov 03 10:29:45.657547 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.657517 5086 generic.go:356] "Generic (PLEG): container finished" podID="369e7292-b645-4eb6-9a5d-62e08910f1ef" containerID="6ec510ef48cec57b5d3572579781a9600c7461a485ccb7a258018d72e7b2069e" exitCode=0 Nov 03 10:29:45.657683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.657582 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" event={"ID":"369e7292-b645-4eb6-9a5d-62e08910f1ef","Type":"ContainerDied","Data":"6ec510ef48cec57b5d3572579781a9600c7461a485ccb7a258018d72e7b2069e"} Nov 03 10:29:45.741214 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.741139 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:29:45.799860 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.799827 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca\") pod \"369e7292-b645-4eb6-9a5d-62e08910f1ef\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " Nov 03 10:29:45.800031 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.799883 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") pod \"369e7292-b645-4eb6-9a5d-62e08910f1ef\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " Nov 03 10:29:45.800031 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.799909 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates\") pod \"369e7292-b645-4eb6-9a5d-62e08910f1ef\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " Nov 03 10:29:45.800031 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.799937 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration\") pod \"369e7292-b645-4eb6-9a5d-62e08910f1ef\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " Nov 03 10:29:45.800031 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.799961 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token\") pod \"369e7292-b645-4eb6-9a5d-62e08910f1ef\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " Nov 03 10:29:45.800260 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.800087 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets\") pod \"369e7292-b645-4eb6-9a5d-62e08910f1ef\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " Nov 03 10:29:45.800260 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.800140 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted\") pod \"369e7292-b645-4eb6-9a5d-62e08910f1ef\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " Nov 03 10:29:45.800260 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.800210 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9\") pod \"369e7292-b645-4eb6-9a5d-62e08910f1ef\" (UID: \"369e7292-b645-4eb6-9a5d-62e08910f1ef\") " Nov 03 10:29:45.800472 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.800316 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca" pod "369e7292-b645-4eb6-9a5d-62e08910f1ef" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:29:45.800472 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.800390 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates" pod "369e7292-b645-4eb6-9a5d-62e08910f1ef" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:29:45.800693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.800640 5086 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-trusted-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:45.800693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.800667 5086 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-certificates\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:45.802310 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.802272 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls" pod "369e7292-b645-4eb6-9a5d-62e08910f1ef" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:29:45.802694 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.802671 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token" pod "369e7292-b645-4eb6-9a5d-62e08910f1ef" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:29:45.802799 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.802772 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration" pod "369e7292-b645-4eb6-9a5d-62e08910f1ef" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:29:45.802862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.802830 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9" pod "369e7292-b645-4eb6-9a5d-62e08910f1ef" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef"). InnerVolumeSpecName "kube-api-access-hlzh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:29:45.803097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.803079 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets" pod "369e7292-b645-4eb6-9a5d-62e08910f1ef" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:29:45.807186 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.807163 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted" pod "369e7292-b645-4eb6-9a5d-62e08910f1ef" (UID: "369e7292-b645-4eb6-9a5d-62e08910f1ef"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:29:45.901576 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.901543 5086 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-image-registry-private-configuration\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:45.901576 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.901570 5086 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-bound-sa-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:45.901576 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.901581 5086 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/369e7292-b645-4eb6-9a5d-62e08910f1ef-installation-pull-secrets\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:45.901796 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.901591 5086 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/369e7292-b645-4eb6-9a5d-62e08910f1ef-ca-trust-extracted\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:45.901796 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.901601 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlzh9\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-kube-api-access-hlzh9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:45.901796 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:45.901614 5086 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/369e7292-b645-4eb6-9a5d-62e08910f1ef-registry-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:29:46.663057 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:46.663018 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" event={"ID":"369e7292-b645-4eb6-9a5d-62e08910f1ef","Type":"ContainerDied","Data":"cd25a392dece585f0a2ecc53e003bc66a2ca00573c4909d42c45e6c42df883a2"} Nov 03 10:29:46.663460 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:46.663066 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678cf4bb67-v4rz9" Nov 03 10:29:46.663460 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:46.663089 5086 scope.go:122] "RemoveContainer" containerID="6ec510ef48cec57b5d3572579781a9600c7461a485ccb7a258018d72e7b2069e" Nov 03 10:29:46.679581 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:46.679559 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678cf4bb67-v4rz9"] Nov 03 10:29:46.682867 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:46.682847 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-678cf4bb67-v4rz9"] Nov 03 10:29:48.168480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:29:48.168450 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369e7292-b645-4eb6-9a5d-62e08910f1ef" path="/var/lib/kubelet/pods/369e7292-b645-4eb6-9a5d-62e08910f1ef/volumes" Nov 03 10:29:54.275251 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:29:54.275214 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:30:24.272729 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:30:24.272686 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:30:25.837960 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:25.837923 5086 generic.go:356] "Generic (PLEG): container finished" podID="e5a3a09a-d7cf-4f67-9b22-99456a86976b" containerID="16a8a1ff97f2227aa5f367c3147b4a4b3b200368a38ebb54500baf507b78e855" exitCode=0 Nov 03 10:30:25.838396 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:25.837996 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f74wv-nz4fj" event={"ID":"e5a3a09a-d7cf-4f67-9b22-99456a86976b","Type":"ContainerDied","Data":"16a8a1ff97f2227aa5f367c3147b4a4b3b200368a38ebb54500baf507b78e855"} Nov 03 10:30:26.995656 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:26.995630 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:30:27.175215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.175127 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-trusted-ca-bundle\") pod \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " Nov 03 10:30:27.175215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.175204 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a3a09a-d7cf-4f67-9b22-99456a86976b-kube-api-access-p86kv\" (UniqueName: \"kubernetes.io/projected/e5a3a09a-d7cf-4f67-9b22-99456a86976b-kube-api-access-p86kv\") pod \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " Nov 03 10:30:27.175432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.175259 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-service-ca-bundle\") pod \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " Nov 03 10:30:27.175432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.175291 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a3a09a-d7cf-4f67-9b22-99456a86976b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5a3a09a-d7cf-4f67-9b22-99456a86976b-archives-path\") pod \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\" (UID: \"e5a3a09a-d7cf-4f67-9b22-99456a86976b\") " Nov 03 10:30:27.175660 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.175631 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-trusted-ca-bundle" pod "e5a3a09a-d7cf-4f67-9b22-99456a86976b" (UID: "e5a3a09a-d7cf-4f67-9b22-99456a86976b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:30:27.175702 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.175638 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-service-ca-bundle" pod "e5a3a09a-d7cf-4f67-9b22-99456a86976b" (UID: "e5a3a09a-d7cf-4f67-9b22-99456a86976b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:30:27.175702 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.175682 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a3a09a-d7cf-4f67-9b22-99456a86976b-archives-path" pod "e5a3a09a-d7cf-4f67-9b22-99456a86976b" (UID: "e5a3a09a-d7cf-4f67-9b22-99456a86976b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:30:27.177465 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.177440 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a3a09a-d7cf-4f67-9b22-99456a86976b-kube-api-access-p86kv" pod "e5a3a09a-d7cf-4f67-9b22-99456a86976b" (UID: "e5a3a09a-d7cf-4f67-9b22-99456a86976b"). InnerVolumeSpecName "kube-api-access-p86kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:30:27.276815 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.276776 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86kv\" (UniqueName: \"kubernetes.io/projected/e5a3a09a-d7cf-4f67-9b22-99456a86976b-kube-api-access-p86kv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:30:27.276815 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.276810 5086 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-service-ca-bundle\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:30:27.276815 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.276821 5086 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e5a3a09a-d7cf-4f67-9b22-99456a86976b-archives-path\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:30:27.277034 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.276830 5086 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5a3a09a-d7cf-4f67-9b22-99456a86976b-trusted-ca-bundle\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:30:27.846707 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.846665 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f74wv-nz4fj" event={"ID":"e5a3a09a-d7cf-4f67-9b22-99456a86976b","Type":"ContainerDied","Data":"05a25239a2f065f6dbf1bd287a0c32726c5f5bce55859ede9e87b62248ea0574"} Nov 03 10:30:27.846707 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.846703 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f74wv-nz4fj" Nov 03 10:30:27.846917 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:30:27.846739 5086 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05a25239a2f065f6dbf1bd287a0c32726c5f5bce55859ede9e87b62248ea0574" Nov 03 10:30:54.272031 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:30:54.271938 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:31:24.273447 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:31:24.273406 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:31:54.272735 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:31:54.272697 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:32:24.273177 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:32:24.273099 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:32:54.272647 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:32:54.272610 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:33:24.271579 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:33:24.271542 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:33:41.449830 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.449796 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw"] Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450111 5086 cpu_manager.go:414] "containerMap: removing container" podUID="0bbacc63-2772-4fa1-a868-afc6bc41fa44" containerName="insights-gathering" Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450122 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbacc63-2772-4fa1-a868-afc6bc41fa44" containerName="insights-gathering" Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450136 5086 cpu_manager.go:414] "containerMap: removing container" podUID="369e7292-b645-4eb6-9a5d-62e08910f1ef" containerName="registry" Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450141 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369e7292-b645-4eb6-9a5d-62e08910f1ef" containerName="registry" Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450150 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c634e6ef-e187-4409-a0a0-bae5793bc137" containerName="registry" Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450155 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c634e6ef-e187-4409-a0a0-bae5793bc137" containerName="registry" Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450164 5086 cpu_manager.go:414] "containerMap: removing container" podUID="e5a3a09a-d7cf-4f67-9b22-99456a86976b" containerName="insights-gathering" Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450168 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a3a09a-d7cf-4f67-9b22-99456a86976b" containerName="insights-gathering" Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450186 5086 cpu_manager.go:414] "containerMap: removing container" podUID="bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" containerName="console" Nov 03 10:33:41.450193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450190 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" containerName="console" Nov 03 10:33:41.450542 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450240 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbacc63-2772-4fa1-a868-afc6bc41fa44" containerName="insights-gathering" Nov 03 10:33:41.450542 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450249 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="369e7292-b645-4eb6-9a5d-62e08910f1ef" containerName="registry" Nov 03 10:33:41.450542 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450254 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c634e6ef-e187-4409-a0a0-bae5793bc137" containerName="registry" Nov 03 10:33:41.450542 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450260 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a3a09a-d7cf-4f67-9b22-99456a86976b" containerName="insights-gathering" Nov 03 10:33:41.450542 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.450270 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0bf7e5-e4e5-4aa3-98b1-d82b82ccceed" containerName="console" Nov 03 10:33:41.453420 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.453404 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.459904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.459777 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-5n4pf\"" Nov 03 10:33:41.461405 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.461386 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 10:33:41.461909 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.461890 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 10:33:41.463389 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.463365 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw"] Nov 03 10:33:41.595039 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.595003 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.595207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.595046 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgmh\" (UniqueName: \"kubernetes.io/projected/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-kube-api-access-szgmh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.595207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.595136 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.695705 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.695672 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.695891 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.695713 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgmh\" (UniqueName: \"kubernetes.io/projected/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-kube-api-access-szgmh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.695891 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.695758 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.696088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.696068 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.696142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.696124 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.703983 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.703913 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgmh\" (UniqueName: \"kubernetes.io/projected/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-kube-api-access-szgmh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:41.769424 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:41.769401 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:33:42.038517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:42.038440 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw"] Nov 03 10:33:42.041407 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:33:42.041378 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1f5bc7_597c_4042_918f_0f83e0ca4f62.slice/crio-1b918a4e4aec7e3a8433bda33c5a79e8b2440931a00dcc6b959b3718676e1bad WatchSource:0}: Error finding container 1b918a4e4aec7e3a8433bda33c5a79e8b2440931a00dcc6b959b3718676e1bad: Status 404 returned error can't find the container with id 1b918a4e4aec7e3a8433bda33c5a79e8b2440931a00dcc6b959b3718676e1bad Nov 03 10:33:42.186697 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:42.186672 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:33:42.188111 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:42.188090 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:33:42.195263 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:42.195240 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:33:42.196621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:42.196605 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:33:42.673545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:42.673504 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" event={"ID":"4c1f5bc7-597c-4042-918f-0f83e0ca4f62","Type":"ContainerStarted","Data":"1b918a4e4aec7e3a8433bda33c5a79e8b2440931a00dcc6b959b3718676e1bad"} Nov 03 10:33:48.703913 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:48.703872 5086 generic.go:356] "Generic (PLEG): container finished" podID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerID="8c6f725adb31bc8470d98bc1d731d2bbeec21037a0d3ac381827163e29deccf9" exitCode=0 Nov 03 10:33:48.704429 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:48.703957 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" event={"ID":"4c1f5bc7-597c-4042-918f-0f83e0ca4f62","Type":"ContainerDied","Data":"8c6f725adb31bc8470d98bc1d731d2bbeec21037a0d3ac381827163e29deccf9"} Nov 03 10:33:48.705151 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:48.705136 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:33:51.718421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:51.718388 5086 generic.go:356] "Generic (PLEG): container finished" podID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerID="30900c490dc827c134fdc500782fd6ee2181f7350e80ae93b79716d955aec277" exitCode=0 Nov 03 10:33:51.718851 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:51.718434 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" event={"ID":"4c1f5bc7-597c-4042-918f-0f83e0ca4f62","Type":"ContainerDied","Data":"30900c490dc827c134fdc500782fd6ee2181f7350e80ae93b79716d955aec277"} Nov 03 10:33:54.276094 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:33:54.276008 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:33:58.755361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:58.755260 5086 generic.go:356] "Generic (PLEG): container finished" podID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerID="eff190aa9d7a4f27dd79ee622cbd40fba69ece32d2fee9b1537165b30d944a15" exitCode=0 Nov 03 10:33:58.755361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:58.755342 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" event={"ID":"4c1f5bc7-597c-4042-918f-0f83e0ca4f62","Type":"ContainerDied","Data":"eff190aa9d7a4f27dd79ee622cbd40fba69ece32d2fee9b1537165b30d944a15"} Nov 03 10:33:59.887286 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:33:59.887262 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:34:00.044553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.044468 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-kube-api-access-szgmh\" (UniqueName: \"kubernetes.io/projected/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-kube-api-access-szgmh\") pod \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " Nov 03 10:34:00.044553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.044525 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-bundle\") pod \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " Nov 03 10:34:00.044804 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.044598 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-util\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-util\") pod \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\" (UID: \"4c1f5bc7-597c-4042-918f-0f83e0ca4f62\") " Nov 03 10:34:00.045051 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.045023 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-bundle" pod "4c1f5bc7-597c-4042-918f-0f83e0ca4f62" (UID: "4c1f5bc7-597c-4042-918f-0f83e0ca4f62"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:00.046840 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.046824 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-kube-api-access-szgmh" pod "4c1f5bc7-597c-4042-918f-0f83e0ca4f62" (UID: "4c1f5bc7-597c-4042-918f-0f83e0ca4f62"). InnerVolumeSpecName "kube-api-access-szgmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:00.049008 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.048984 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-util" pod "4c1f5bc7-597c-4042-918f-0f83e0ca4f62" (UID: "4c1f5bc7-597c-4042-918f-0f83e0ca4f62"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:00.146062 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.146032 5086 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-util\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:34:00.146062 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.146058 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgmh\" (UniqueName: \"kubernetes.io/projected/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-kube-api-access-szgmh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:34:00.146062 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.146069 5086 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1f5bc7-597c-4042-918f-0f83e0ca4f62-bundle\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:34:00.766080 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.766048 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" event={"ID":"4c1f5bc7-597c-4042-918f-0f83e0ca4f62","Type":"ContainerDied","Data":"1b918a4e4aec7e3a8433bda33c5a79e8b2440931a00dcc6b959b3718676e1bad"} Nov 03 10:34:00.766080 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.766069 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckrcrw" Nov 03 10:34:00.766291 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:00.766098 5086 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b918a4e4aec7e3a8433bda33c5a79e8b2440931a00dcc6b959b3718676e1bad" Nov 03 10:34:24.271996 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:34:24.271957 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:34:33.072408 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.072369 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx"] Nov 03 10:34:33.072878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.072706 5086 cpu_manager.go:414] "containerMap: removing container" podUID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerName="pull" Nov 03 10:34:33.072878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.072720 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerName="pull" Nov 03 10:34:33.072878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.072741 5086 cpu_manager.go:414] "containerMap: removing container" podUID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerName="extract" Nov 03 10:34:33.072878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.072746 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerName="extract" Nov 03 10:34:33.072878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.072770 5086 cpu_manager.go:414] "containerMap: removing container" podUID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerName="util" Nov 03 10:34:33.072878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.072775 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerName="util" Nov 03 10:34:33.072878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.072836 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1f5bc7-597c-4042-918f-0f83e0ca4f62" containerName="extract" Nov 03 10:34:33.076096 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.076079 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.083557 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.083534 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 10:34:33.088054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.088031 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 10:34:33.088159 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.088084 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 10:34:33.088159 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.088105 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 10:34:33.100616 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.100584 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx"] Nov 03 10:34:33.188307 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.188278 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-s78mx\"" Nov 03 10:34:33.188517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.188501 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 10:34:33.206874 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.206848 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f693aed2-4254-4921-bcdc-cc95631d17c1-manifests\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.206968 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.206887 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f693aed2-4254-4921-bcdc-cc95631d17c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.206968 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.206920 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mgz2\" (UniqueName: \"kubernetes.io/projected/f693aed2-4254-4921-bcdc-cc95631d17c1-kube-api-access-2mgz2\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.207053 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.207022 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f693aed2-4254-4921-bcdc-cc95631d17c1-tmp\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.207090 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.207079 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f693aed2-4254-4921-bcdc-cc95631d17c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.207120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.207112 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f693aed2-4254-4921-bcdc-cc95631d17c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.307981 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.307950 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f693aed2-4254-4921-bcdc-cc95631d17c1-manifests\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.308154 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.308003 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f693aed2-4254-4921-bcdc-cc95631d17c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.308238 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.308212 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mgz2\" (UniqueName: \"kubernetes.io/projected/f693aed2-4254-4921-bcdc-cc95631d17c1-kube-api-access-2mgz2\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.308349 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.308333 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f693aed2-4254-4921-bcdc-cc95631d17c1-tmp\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.308455 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.308435 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f693aed2-4254-4921-bcdc-cc95631d17c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.308534 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.308510 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f693aed2-4254-4921-bcdc-cc95631d17c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.308752 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.308730 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f693aed2-4254-4921-bcdc-cc95631d17c1-manifests\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.310561 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.310532 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f693aed2-4254-4921-bcdc-cc95631d17c1-tmp\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.310682 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.310638 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f693aed2-4254-4921-bcdc-cc95631d17c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.310835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.310817 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f693aed2-4254-4921-bcdc-cc95631d17c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.310933 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.310909 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f693aed2-4254-4921-bcdc-cc95631d17c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.316949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.316928 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mgz2\" (UniqueName: \"kubernetes.io/projected/f693aed2-4254-4921-bcdc-cc95631d17c1-kube-api-access-2mgz2\") pod \"opendatahub-operator-controller-manager-6688c64986-ftpqx\" (UID: \"f693aed2-4254-4921-bcdc-cc95631d17c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.390704 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.390681 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:33.686339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.686232 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx"] Nov 03 10:34:33.719520 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.719493 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k95vx"] Nov 03 10:34:33.724190 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.724169 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" Nov 03 10:34:33.738441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.738412 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 10:34:33.738545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.738502 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qc6qg\"" Nov 03 10:34:33.738545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.738510 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 10:34:33.740385 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.740365 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k95vx"] Nov 03 10:34:33.813264 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.813232 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2s5z\" (UniqueName: \"kubernetes.io/projected/09b8b602-a32b-4d2e-ae58-d4d15c56457e-kube-api-access-p2s5z\") pod \"authorino-operator-7b7bb4bc77-k95vx\" (UID: \"09b8b602-a32b-4d2e-ae58-d4d15c56457e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" Nov 03 10:34:33.911184 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.911148 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" event={"ID":"f693aed2-4254-4921-bcdc-cc95631d17c1","Type":"ContainerStarted","Data":"cca9bb30bd8da5c501f88c3dcfce18995a8bfa2fb7334310d593215597a03dc4"} Nov 03 10:34:33.914529 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.914508 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2s5z\" (UniqueName: \"kubernetes.io/projected/09b8b602-a32b-4d2e-ae58-d4d15c56457e-kube-api-access-p2s5z\") pod \"authorino-operator-7b7bb4bc77-k95vx\" (UID: \"09b8b602-a32b-4d2e-ae58-d4d15c56457e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" Nov 03 10:34:33.925663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:33.925635 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2s5z\" (UniqueName: \"kubernetes.io/projected/09b8b602-a32b-4d2e-ae58-d4d15c56457e-kube-api-access-p2s5z\") pod \"authorino-operator-7b7bb4bc77-k95vx\" (UID: \"09b8b602-a32b-4d2e-ae58-d4d15c56457e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" Nov 03 10:34:34.036787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:34.036710 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" Nov 03 10:34:34.307566 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:34.307483 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k95vx"] Nov 03 10:34:34.310766 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:34:34.310738 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b8b602_a32b_4d2e_ae58_d4d15c56457e.slice/crio-3bfaf3df0f62fff9d3805fa3ecc78f6286ef29d6d4a505dbb042c75fe71df839 WatchSource:0}: Error finding container 3bfaf3df0f62fff9d3805fa3ecc78f6286ef29d6d4a505dbb042c75fe71df839: Status 404 returned error can't find the container with id 3bfaf3df0f62fff9d3805fa3ecc78f6286ef29d6d4a505dbb042c75fe71df839 Nov 03 10:34:34.917509 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:34.917466 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" event={"ID":"09b8b602-a32b-4d2e-ae58-d4d15c56457e","Type":"ContainerStarted","Data":"3bfaf3df0f62fff9d3805fa3ecc78f6286ef29d6d4a505dbb042c75fe71df839"} Nov 03 10:34:37.936673 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:37.936586 5086 generic.go:356] "Generic (PLEG): container finished" podID="f693aed2-4254-4921-bcdc-cc95631d17c1" containerID="f71a52544a1b1a85fd8507a68dd61e8ef7c0dce31bb3edd0f4564c8e74dd116f" exitCode=0 Nov 03 10:34:37.937048 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:37.936672 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" event={"ID":"f693aed2-4254-4921-bcdc-cc95631d17c1","Type":"ContainerDied","Data":"f71a52544a1b1a85fd8507a68dd61e8ef7c0dce31bb3edd0f4564c8e74dd116f"} Nov 03 10:34:37.938540 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:37.938516 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" event={"ID":"09b8b602-a32b-4d2e-ae58-d4d15c56457e","Type":"ContainerStarted","Data":"bb69d1714e3e35fc4f2d069ffc20e6e76c5ab84799980955081065b6ad7ca9ac"} Nov 03 10:34:37.938606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:37.938582 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" Nov 03 10:34:38.017030 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:38.016980 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" podStartSLOduration=2.0380330190000002 podStartE2EDuration="5.016966398s" totalImagesPullingTime="2.978933379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:38.015025736 +0000 UTC m=+355.999897021" watchObservedRunningTime="2026-11-03 10:34:38.016966398 +0000 UTC m=+356.001837681" Nov 03 10:34:38.945023 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:38.944986 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" event={"ID":"f693aed2-4254-4921-bcdc-cc95631d17c1","Type":"ContainerStarted","Data":"7a5aa2e6e7126251f525aa46a480cbc66d0fa468dfd8914db37d4eb7c49f6ef9"} Nov 03 10:34:38.945556 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:38.945200 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:38.989065 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:38.989015 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" podStartSLOduration=1.7142729060000002 podStartE2EDuration="5.989001812s" totalImagesPullingTime="3.736643082s" totalInitContainerRuntime="538.085824ms" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:38.987817043 +0000 UTC m=+356.972688338" watchObservedRunningTime="2026-11-03 10:34:38.989001812 +0000 UTC m=+356.973873094" Nov 03 10:34:41.415784 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.415750 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk"] Nov 03 10:34:41.419325 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.419306 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:34:41.423257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.423233 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cxm7r\"" Nov 03 10:34:41.431364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.431335 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk"] Nov 03 10:34:41.586680 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.586640 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgg9\" (UniqueName: \"kubernetes.io/projected/2ed321b2-0274-4440-9189-96c520236247-kube-api-access-8qgg9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8whrk\" (UID: \"2ed321b2-0274-4440-9189-96c520236247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:34:41.586894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.586787 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed321b2-0274-4440-9189-96c520236247-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8whrk\" (UID: \"2ed321b2-0274-4440-9189-96c520236247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:34:41.687738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.687642 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgg9\" (UniqueName: \"kubernetes.io/projected/2ed321b2-0274-4440-9189-96c520236247-kube-api-access-8qgg9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8whrk\" (UID: \"2ed321b2-0274-4440-9189-96c520236247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:34:41.687738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.687744 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed321b2-0274-4440-9189-96c520236247-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8whrk\" (UID: \"2ed321b2-0274-4440-9189-96c520236247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:34:41.688097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.688079 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed321b2-0274-4440-9189-96c520236247-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8whrk\" (UID: \"2ed321b2-0274-4440-9189-96c520236247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:34:41.695828 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.695806 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgg9\" (UniqueName: \"kubernetes.io/projected/2ed321b2-0274-4440-9189-96c520236247-kube-api-access-8qgg9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8whrk\" (UID: \"2ed321b2-0274-4440-9189-96c520236247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:34:41.731785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:41.731743 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:34:42.009829 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:42.009799 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk"] Nov 03 10:34:42.012440 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:34:42.012412 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed321b2_0274_4440_9189_96c520236247.slice/crio-8ca5834996293536ca0a611842d3f1348d287823bb2a458189e6a23bf7ba716b WatchSource:0}: Error finding container 8ca5834996293536ca0a611842d3f1348d287823bb2a458189e6a23bf7ba716b: Status 404 returned error can't find the container with id 8ca5834996293536ca0a611842d3f1348d287823bb2a458189e6a23bf7ba716b Nov 03 10:34:42.963006 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:42.962971 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" event={"ID":"2ed321b2-0274-4440-9189-96c520236247","Type":"ContainerStarted","Data":"8ca5834996293536ca0a611842d3f1348d287823bb2a458189e6a23bf7ba716b"} Nov 03 10:34:46.983821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:46.983779 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" event={"ID":"2ed321b2-0274-4440-9189-96c520236247","Type":"ContainerStarted","Data":"54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8"} Nov 03 10:34:46.984204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:46.983922 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:34:47.011744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:47.011683 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" podStartSLOduration=1.465228292 podStartE2EDuration="6.011665624s" totalImagesPullingTime="4.546437332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:47.00958307 +0000 UTC m=+364.994454344" watchObservedRunningTime="2026-11-03 10:34:47.011665624 +0000 UTC m=+364.996536907" Nov 03 10:34:48.947852 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:48.947814 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k95vx" Nov 03 10:34:49.951560 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:49.951526 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-ftpqx" Nov 03 10:34:50.438345 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:50.438311 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg"] Nov 03 10:34:50.442132 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:50.442111 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" Nov 03 10:34:50.450155 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:50.450134 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-b9vdr\"" Nov 03 10:34:50.458403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:50.458363 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg"] Nov 03 10:34:50.473903 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:50.473871 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbfm\" (UniqueName: \"kubernetes.io/projected/578a0ec5-eda4-4576-acd8-8cd12251d4dd-kube-api-access-jmbfm\") pod \"limitador-operator-controller-manager-5778bb4449-4shtg\" (UID: \"578a0ec5-eda4-4576-acd8-8cd12251d4dd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" Nov 03 10:34:50.574420 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:50.574381 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbfm\" (UniqueName: \"kubernetes.io/projected/578a0ec5-eda4-4576-acd8-8cd12251d4dd-kube-api-access-jmbfm\") pod \"limitador-operator-controller-manager-5778bb4449-4shtg\" (UID: \"578a0ec5-eda4-4576-acd8-8cd12251d4dd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" Nov 03 10:34:50.586332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:50.586286 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbfm\" (UniqueName: \"kubernetes.io/projected/578a0ec5-eda4-4576-acd8-8cd12251d4dd-kube-api-access-jmbfm\") pod \"limitador-operator-controller-manager-5778bb4449-4shtg\" (UID: \"578a0ec5-eda4-4576-acd8-8cd12251d4dd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" Nov 03 10:34:50.754422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:50.754333 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" Nov 03 10:34:51.028606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:51.028582 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg"] Nov 03 10:34:51.030974 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:34:51.030946 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578a0ec5_eda4_4576_acd8_8cd12251d4dd.slice/crio-69f85d9ac7152477ba5db55f9c570cc8b3fadd4a1ec3a81d994034ebfda7f030 WatchSource:0}: Error finding container 69f85d9ac7152477ba5db55f9c570cc8b3fadd4a1ec3a81d994034ebfda7f030: Status 404 returned error can't find the container with id 69f85d9ac7152477ba5db55f9c570cc8b3fadd4a1ec3a81d994034ebfda7f030 Nov 03 10:34:52.009450 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:52.009415 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" event={"ID":"578a0ec5-eda4-4576-acd8-8cd12251d4dd","Type":"ContainerStarted","Data":"69f85d9ac7152477ba5db55f9c570cc8b3fadd4a1ec3a81d994034ebfda7f030"} Nov 03 10:34:53.021568 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:53.021472 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" event={"ID":"578a0ec5-eda4-4576-acd8-8cd12251d4dd","Type":"ContainerStarted","Data":"41c3b54e538729535686917e25b828fd7c664fba097c1327f6f80aedeb03f5c9"} Nov 03 10:34:53.021958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:53.021748 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" Nov 03 10:34:53.039615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:53.039555 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" podStartSLOduration=1.350062278 podStartE2EDuration="3.03953793s" totalImagesPullingTime="1.689475652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:53.038826487 +0000 UTC m=+371.023697769" watchObservedRunningTime="2026-11-03 10:34:53.03953793 +0000 UTC m=+371.024409214" Nov 03 10:34:54.273240 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:34:54.273197 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:34:57.991387 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:34:57.991352 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:35:04.028202 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:04.028169 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4shtg" Nov 03 10:35:06.246395 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:06.246358 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wmm6t"] Nov 03 10:35:06.252829 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:06.252805 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" Nov 03 10:35:06.256453 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:06.256425 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wmm6t"] Nov 03 10:35:06.259210 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:06.259186 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wxmwr\"" Nov 03 10:35:06.287246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:06.287215 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pm6\" (UniqueName: \"kubernetes.io/projected/c0096830-293d-4de0-9199-6de3b89b8d0b-kube-api-access-26pm6\") pod \"authorino-57f7f8659f-wmm6t\" (UID: \"c0096830-293d-4de0-9199-6de3b89b8d0b\") " pod="kuadrant-system/authorino-57f7f8659f-wmm6t" Nov 03 10:35:06.388487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:06.388449 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pm6\" (UniqueName: \"kubernetes.io/projected/c0096830-293d-4de0-9199-6de3b89b8d0b-kube-api-access-26pm6\") pod \"authorino-57f7f8659f-wmm6t\" (UID: \"c0096830-293d-4de0-9199-6de3b89b8d0b\") " pod="kuadrant-system/authorino-57f7f8659f-wmm6t" Nov 03 10:35:06.395805 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:06.395777 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pm6\" (UniqueName: \"kubernetes.io/projected/c0096830-293d-4de0-9199-6de3b89b8d0b-kube-api-access-26pm6\") pod \"authorino-57f7f8659f-wmm6t\" (UID: \"c0096830-293d-4de0-9199-6de3b89b8d0b\") " pod="kuadrant-system/authorino-57f7f8659f-wmm6t" Nov 03 10:35:06.578233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:06.578199 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" Nov 03 10:35:06.850641 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:06.850557 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wmm6t"] Nov 03 10:35:06.856733 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:35:06.856701 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0096830_293d_4de0_9199_6de3b89b8d0b.slice/crio-74623013f0fbdd7cb92552b747920e17c3eb40481748d84c2620d3e2af65bbe3 WatchSource:0}: Error finding container 74623013f0fbdd7cb92552b747920e17c3eb40481748d84c2620d3e2af65bbe3: Status 404 returned error can't find the container with id 74623013f0fbdd7cb92552b747920e17c3eb40481748d84c2620d3e2af65bbe3 Nov 03 10:35:07.087341 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:07.087284 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" event={"ID":"c0096830-293d-4de0-9199-6de3b89b8d0b","Type":"ContainerStarted","Data":"74623013f0fbdd7cb92552b747920e17c3eb40481748d84c2620d3e2af65bbe3"} Nov 03 10:35:08.234583 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.234549 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657"] Nov 03 10:35:08.238263 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.238243 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.241714 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.241688 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 10:35:08.242785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.242768 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 10:35:08.243008 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.242991 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-9tbvb\"" Nov 03 10:35:08.245986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.245861 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 10:35:08.247683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.247658 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657"] Nov 03 10:35:08.303356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.303322 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f779485f-1f41-481a-9fb8-4ea051dfa92f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.303524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.303376 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f779485f-1f41-481a-9fb8-4ea051dfa92f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.303524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.303478 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f779485f-1f41-481a-9fb8-4ea051dfa92f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.303524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.303518 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6fdp\" (UniqueName: \"kubernetes.io/projected/f779485f-1f41-481a-9fb8-4ea051dfa92f-kube-api-access-q6fdp\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.404777 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.404745 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f779485f-1f41-481a-9fb8-4ea051dfa92f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.404777 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.404782 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f779485f-1f41-481a-9fb8-4ea051dfa92f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.405058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.404812 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f779485f-1f41-481a-9fb8-4ea051dfa92f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.405058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.404840 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6fdp\" (UniqueName: \"kubernetes.io/projected/f779485f-1f41-481a-9fb8-4ea051dfa92f-kube-api-access-q6fdp\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.405167 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.405151 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f779485f-1f41-481a-9fb8-4ea051dfa92f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.405288 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.405263 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f779485f-1f41-481a-9fb8-4ea051dfa92f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.405735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.405710 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f779485f-1f41-481a-9fb8-4ea051dfa92f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.413061 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.413032 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6fdp\" (UniqueName: \"kubernetes.io/projected/f779485f-1f41-481a-9fb8-4ea051dfa92f-kube-api-access-q6fdp\") pod \"kserve-module-controller-manager-7566cf5fc8-c2657\" (UID: \"f779485f-1f41-481a-9fb8-4ea051dfa92f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.553354 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.553266 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:08.835162 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:08.834984 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657"] Nov 03 10:35:08.837674 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:35:08.837647 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf779485f_1f41_481a_9fb8_4ea051dfa92f.slice/crio-aba0c0c23e0c2636d644e3ec863cfdf173617fee9134a2ea4e1bb543fe1f6748 WatchSource:0}: Error finding container aba0c0c23e0c2636d644e3ec863cfdf173617fee9134a2ea4e1bb543fe1f6748: Status 404 returned error can't find the container with id aba0c0c23e0c2636d644e3ec863cfdf173617fee9134a2ea4e1bb543fe1f6748 Nov 03 10:35:09.099637 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:09.099544 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" event={"ID":"f779485f-1f41-481a-9fb8-4ea051dfa92f","Type":"ContainerStarted","Data":"aba0c0c23e0c2636d644e3ec863cfdf173617fee9134a2ea4e1bb543fe1f6748"} Nov 03 10:35:09.101004 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:09.100971 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" event={"ID":"c0096830-293d-4de0-9199-6de3b89b8d0b","Type":"ContainerStarted","Data":"0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39"} Nov 03 10:35:09.115391 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:09.115343 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" podStartSLOduration=1.087112605 podStartE2EDuration="3.115330158s" totalImagesPullingTime="2.028217553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:09.114072024 +0000 UTC m=+387.098943307" watchObservedRunningTime="2026-11-03 10:35:09.115330158 +0000 UTC m=+387.100201441" Nov 03 10:35:12.118503 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:12.118466 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" event={"ID":"f779485f-1f41-481a-9fb8-4ea051dfa92f","Type":"ContainerStarted","Data":"024c8f3f3b4f353aca4eb93b22d5f664dc6a386af292e38bc6932e4ed03beccd"} Nov 03 10:35:12.118882 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:12.118644 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:12.138053 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:12.138004 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" podStartSLOduration=1.224870856 podStartE2EDuration="4.137991257s" totalImagesPullingTime="2.913120401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:12.135724519 +0000 UTC m=+390.120595803" watchObservedRunningTime="2026-11-03 10:35:12.137991257 +0000 UTC m=+390.122862577" Nov 03 10:35:15.668080 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:15.668046 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk"] Nov 03 10:35:15.668592 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:15.668321 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" podUID="2ed321b2-0274-4440-9189-96c520236247" containerName="manager" containerID="cri-o://54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8" gracePeriod=10 Nov 03 10:35:15.909223 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:15.909199 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:35:15.969127 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:15.969047 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed321b2-0274-4440-9189-96c520236247-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed321b2-0274-4440-9189-96c520236247-extensions-socket-volume\") pod \"2ed321b2-0274-4440-9189-96c520236247\" (UID: \"2ed321b2-0274-4440-9189-96c520236247\") " Nov 03 10:35:15.969274 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:15.969213 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed321b2-0274-4440-9189-96c520236247-kube-api-access-8qgg9\" (UniqueName: \"kubernetes.io/projected/2ed321b2-0274-4440-9189-96c520236247-kube-api-access-8qgg9\") pod \"2ed321b2-0274-4440-9189-96c520236247\" (UID: \"2ed321b2-0274-4440-9189-96c520236247\") " Nov 03 10:35:15.969625 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:15.969604 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed321b2-0274-4440-9189-96c520236247-extensions-socket-volume" pod "2ed321b2-0274-4440-9189-96c520236247" (UID: "2ed321b2-0274-4440-9189-96c520236247"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:35:15.971506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:15.971481 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed321b2-0274-4440-9189-96c520236247-kube-api-access-8qgg9" pod "2ed321b2-0274-4440-9189-96c520236247" (UID: "2ed321b2-0274-4440-9189-96c520236247"). InnerVolumeSpecName "kube-api-access-8qgg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:35:16.070155 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.070117 5086 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ed321b2-0274-4440-9189-96c520236247-extensions-socket-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:35:16.070155 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.070146 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgg9\" (UniqueName: \"kubernetes.io/projected/2ed321b2-0274-4440-9189-96c520236247-kube-api-access-8qgg9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:35:16.139394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.139360 5086 generic.go:356] "Generic (PLEG): container finished" podID="2ed321b2-0274-4440-9189-96c520236247" containerID="54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8" exitCode=0 Nov 03 10:35:16.139544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.139407 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" Nov 03 10:35:16.139544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.139419 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" event={"ID":"2ed321b2-0274-4440-9189-96c520236247","Type":"ContainerDied","Data":"54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8"} Nov 03 10:35:16.139544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.139462 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk" event={"ID":"2ed321b2-0274-4440-9189-96c520236247","Type":"ContainerDied","Data":"8ca5834996293536ca0a611842d3f1348d287823bb2a458189e6a23bf7ba716b"} Nov 03 10:35:16.139544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.139492 5086 scope.go:122] "RemoveContainer" containerID="54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8" Nov 03 10:35:16.148781 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.148763 5086 scope.go:122] "RemoveContainer" containerID="54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8" Nov 03 10:35:16.149065 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:35:16.149046 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8\": container with ID starting with 54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8 not found: ID does not exist" containerID="54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8" Nov 03 10:35:16.149118 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.149076 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8"} err="failed to get container status \"54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8\": rpc error: code = NotFound desc = could not find container \"54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8\": container with ID starting with 54541f99628fce0d7e089b247be6df9f3456fae08bd0323cfe8f2514f84d1ed8 not found: ID does not exist" Nov 03 10:35:16.171426 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.171396 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk"] Nov 03 10:35:16.179875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:16.179848 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8whrk"] Nov 03 10:35:18.171040 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:18.171006 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed321b2-0274-4440-9189-96c520236247" path="/var/lib/kubelet/pods/2ed321b2-0274-4440-9189-96c520236247/volumes" Nov 03 10:35:24.272202 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:35:24.272121 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:35:25.793273 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.793234 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426"] Nov 03 10:35:25.793704 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.793638 5086 cpu_manager.go:414] "containerMap: removing container" podUID="2ed321b2-0274-4440-9189-96c520236247" containerName="manager" Nov 03 10:35:25.793704 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.793650 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed321b2-0274-4440-9189-96c520236247" containerName="manager" Nov 03 10:35:25.793785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.793729 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed321b2-0274-4440-9189-96c520236247" containerName="manager" Nov 03 10:35:25.796767 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.796748 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.804048 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.804027 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 10:35:25.804276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.804118 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 10:35:25.804276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.804106 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 10:35:25.804276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.804245 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 10:35:25.804276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.804199 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-xqw6n\"" Nov 03 10:35:25.804553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.804531 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 10:35:25.806473 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.806451 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426"] Nov 03 10:35:25.860952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.860917 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-manager-config\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.860952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.860956 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2w4\" (UniqueName: \"kubernetes.io/projected/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-kube-api-access-4r2w4\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.861137 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.860984 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-cert\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.861137 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.861075 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-metrics-certs\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.962042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.962005 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-cert\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.962209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.962050 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-metrics-certs\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.962209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.962101 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-manager-config\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.962209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.962122 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2w4\" (UniqueName: \"kubernetes.io/projected/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-kube-api-access-4r2w4\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.962845 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.962819 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-manager-config\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.964580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.964561 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-cert\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.964661 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.964632 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-metrics-certs\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:25.970197 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:25.970176 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2w4\" (UniqueName: \"kubernetes.io/projected/9b3b832d-0518-44a0-96ea-a99fbbfd2df8-kube-api-access-4r2w4\") pod \"jobset-controller-manager-664b6d8b68-22426\" (UID: \"9b3b832d-0518-44a0-96ea-a99fbbfd2df8\") " pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:26.112614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:26.112564 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:26.384133 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:26.384107 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426"] Nov 03 10:35:26.386622 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:35:26.386598 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3b832d_0518_44a0_96ea_a99fbbfd2df8.slice/crio-de0a42818fd51d94414592a766092993f45cf224f128b6c467c6ebb5d5decb3a WatchSource:0}: Error finding container de0a42818fd51d94414592a766092993f45cf224f128b6c467c6ebb5d5decb3a: Status 404 returned error can't find the container with id de0a42818fd51d94414592a766092993f45cf224f128b6c467c6ebb5d5decb3a Nov 03 10:35:27.200658 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:27.200626 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" event={"ID":"9b3b832d-0518-44a0-96ea-a99fbbfd2df8","Type":"ContainerStarted","Data":"de0a42818fd51d94414592a766092993f45cf224f128b6c467c6ebb5d5decb3a"} Nov 03 10:35:30.216305 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:30.216269 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" event={"ID":"9b3b832d-0518-44a0-96ea-a99fbbfd2df8","Type":"ContainerStarted","Data":"991e76c35d16437162f908c85a68b31b39878cfea0e31b5d569ac943dc9ad699"} Nov 03 10:35:30.216734 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:30.216403 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:30.233774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:30.233723 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" podStartSLOduration=2.052910295 podStartE2EDuration="5.233708473s" totalImagesPullingTime="3.180798178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:30.231707346 +0000 UTC m=+408.216578630" watchObservedRunningTime="2026-11-03 10:35:30.233708473 +0000 UTC m=+408.218579810" Nov 03 10:35:33.125324 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:33.125265 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c2657" Nov 03 10:35:35.004729 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:35.004690 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lcg5t"] Nov 03 10:35:35.008535 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:35.008506 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" Nov 03 10:35:35.014989 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:35.014962 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lcg5t"] Nov 03 10:35:35.141016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:35.140979 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdjv\" (UniqueName: \"kubernetes.io/projected/8e45a9db-16bd-4e4b-87d1-e09926df4a75-kube-api-access-zmdjv\") pod \"authorino-56c6c9ddd5-lcg5t\" (UID: \"8e45a9db-16bd-4e4b-87d1-e09926df4a75\") " pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" Nov 03 10:35:35.242180 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:35.242148 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdjv\" (UniqueName: \"kubernetes.io/projected/8e45a9db-16bd-4e4b-87d1-e09926df4a75-kube-api-access-zmdjv\") pod \"authorino-56c6c9ddd5-lcg5t\" (UID: \"8e45a9db-16bd-4e4b-87d1-e09926df4a75\") " pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" Nov 03 10:35:35.249767 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:35.249738 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdjv\" (UniqueName: \"kubernetes.io/projected/8e45a9db-16bd-4e4b-87d1-e09926df4a75-kube-api-access-zmdjv\") pod \"authorino-56c6c9ddd5-lcg5t\" (UID: \"8e45a9db-16bd-4e4b-87d1-e09926df4a75\") " pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" Nov 03 10:35:35.322871 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:35.322834 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" Nov 03 10:35:35.596882 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:35.596856 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lcg5t"] Nov 03 10:35:35.599249 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:35:35.599223 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e45a9db_16bd_4e4b_87d1_e09926df4a75.slice/crio-c4b88572cdbe0bcb8e7b1d3a09043b20a0765c0bbdb91e9732bf94c24455fb57 WatchSource:0}: Error finding container c4b88572cdbe0bcb8e7b1d3a09043b20a0765c0bbdb91e9732bf94c24455fb57: Status 404 returned error can't find the container with id c4b88572cdbe0bcb8e7b1d3a09043b20a0765c0bbdb91e9732bf94c24455fb57 Nov 03 10:35:36.246753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:36.246714 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" event={"ID":"8e45a9db-16bd-4e4b-87d1-e09926df4a75","Type":"ContainerStarted","Data":"c4b88572cdbe0bcb8e7b1d3a09043b20a0765c0bbdb91e9732bf94c24455fb57"} Nov 03 10:35:37.253236 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:37.253203 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" event={"ID":"8e45a9db-16bd-4e4b-87d1-e09926df4a75","Type":"ContainerStarted","Data":"989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9"} Nov 03 10:35:37.267525 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:37.267480 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" podStartSLOduration=2.443080728 podStartE2EDuration="3.26746634s" totalImagesPullingTime="824.385612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:37.266235671 +0000 UTC m=+415.251106955" watchObservedRunningTime="2026-11-03 10:35:37.26746634 +0000 UTC m=+415.252337623" Nov 03 10:35:37.290503 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:37.290467 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wmm6t"] Nov 03 10:35:37.290726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:37.290697 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" podUID="c0096830-293d-4de0-9199-6de3b89b8d0b" containerName="authorino" containerID="cri-o://0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39" gracePeriod=30 Nov 03 10:35:37.554203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:37.554176 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" Nov 03 10:35:37.669037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:37.669002 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0096830-293d-4de0-9199-6de3b89b8d0b-kube-api-access-26pm6\" (UniqueName: \"kubernetes.io/projected/c0096830-293d-4de0-9199-6de3b89b8d0b-kube-api-access-26pm6\") pod \"c0096830-293d-4de0-9199-6de3b89b8d0b\" (UID: \"c0096830-293d-4de0-9199-6de3b89b8d0b\") " Nov 03 10:35:37.671064 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:37.671032 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0096830-293d-4de0-9199-6de3b89b8d0b-kube-api-access-26pm6" pod "c0096830-293d-4de0-9199-6de3b89b8d0b" (UID: "c0096830-293d-4de0-9199-6de3b89b8d0b"). InnerVolumeSpecName "kube-api-access-26pm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:35:37.769810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:37.769766 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26pm6\" (UniqueName: \"kubernetes.io/projected/c0096830-293d-4de0-9199-6de3b89b8d0b-kube-api-access-26pm6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:35:38.259563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:38.259530 5086 generic.go:356] "Generic (PLEG): container finished" podID="c0096830-293d-4de0-9199-6de3b89b8d0b" containerID="0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39" exitCode=0 Nov 03 10:35:38.259932 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:38.259849 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" Nov 03 10:35:38.260142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:38.260117 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" event={"ID":"c0096830-293d-4de0-9199-6de3b89b8d0b","Type":"ContainerDied","Data":"0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39"} Nov 03 10:35:38.260185 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:38.260165 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wmm6t" event={"ID":"c0096830-293d-4de0-9199-6de3b89b8d0b","Type":"ContainerDied","Data":"74623013f0fbdd7cb92552b747920e17c3eb40481748d84c2620d3e2af65bbe3"} Nov 03 10:35:38.260218 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:38.260190 5086 scope.go:122] "RemoveContainer" containerID="0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39" Nov 03 10:35:38.268803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:38.268785 5086 scope.go:122] "RemoveContainer" containerID="0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39" Nov 03 10:35:38.269078 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:35:38.269057 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39\": container with ID starting with 0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39 not found: ID does not exist" containerID="0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39" Nov 03 10:35:38.269126 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:38.269089 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39"} err="failed to get container status \"0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39\": rpc error: code = NotFound desc = could not find container \"0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39\": container with ID starting with 0b4ff3ace4b6425a60aea4e3dae518bf53f0119a835eebf73b2af56319b03e39 not found: ID does not exist" Nov 03 10:35:38.277754 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:38.277727 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wmm6t"] Nov 03 10:35:38.278945 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:38.278927 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wmm6t"] Nov 03 10:35:40.168583 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:40.168541 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0096830-293d-4de0-9199-6de3b89b8d0b" path="/var/lib/kubelet/pods/c0096830-293d-4de0-9199-6de3b89b8d0b/volumes" Nov 03 10:35:41.227745 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:35:41.227715 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-664b6d8b68-22426" Nov 03 10:35:54.273935 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:35:54.273898 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:36:01.160521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.160404 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-kjdj9"] Nov 03 10:36:01.161565 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.161538 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c0096830-293d-4de0-9199-6de3b89b8d0b" containerName="authorino" Nov 03 10:36:01.161565 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.161564 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0096830-293d-4de0-9199-6de3b89b8d0b" containerName="authorino" Nov 03 10:36:01.161728 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.161673 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0096830-293d-4de0-9199-6de3b89b8d0b" containerName="authorino" Nov 03 10:36:01.168681 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.168658 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:01.178482 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.176660 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-kjdj9"] Nov 03 10:36:01.191811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.191775 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-8lglj"] Nov 03 10:36:01.196366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.196333 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.202830 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.202799 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8lglj"] Nov 03 10:36:01.203823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.203800 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-6l7sl\"" Nov 03 10:36:01.203945 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.203882 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 10:36:01.222671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.222625 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 10:36:01.222860 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.222755 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-58dz2\"" Nov 03 10:36:01.222860 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.222816 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 10:36:01.278765 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.278728 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45pb5\" (UniqueName: \"kubernetes.io/projected/d804b100-c032-4251-8baf-c40f1e01f919-kube-api-access-45pb5\") pod \"model-serving-api-75bd89fdfc-kjdj9\" (UID: \"d804b100-c032-4251-8baf-c40f1e01f919\") " pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:01.278949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.278783 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af95fda-8dff-495e-8d59-334e7aeca7f8-cert\") pod \"odh-model-controller-769875db56-8lglj\" (UID: \"1af95fda-8dff-495e-8d59-334e7aeca7f8\") " pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.278949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.278897 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvlx\" (UniqueName: \"kubernetes.io/projected/1af95fda-8dff-495e-8d59-334e7aeca7f8-kube-api-access-6lvlx\") pod \"odh-model-controller-769875db56-8lglj\" (UID: \"1af95fda-8dff-495e-8d59-334e7aeca7f8\") " pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.278949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.278937 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af95fda-8dff-495e-8d59-334e7aeca7f8-metrics-certs\") pod \"odh-model-controller-769875db56-8lglj\" (UID: \"1af95fda-8dff-495e-8d59-334e7aeca7f8\") " pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.279072 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.278966 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d804b100-c032-4251-8baf-c40f1e01f919-tls-certs\") pod \"model-serving-api-75bd89fdfc-kjdj9\" (UID: \"d804b100-c032-4251-8baf-c40f1e01f919\") " pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:01.379643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.379609 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45pb5\" (UniqueName: \"kubernetes.io/projected/d804b100-c032-4251-8baf-c40f1e01f919-kube-api-access-45pb5\") pod \"model-serving-api-75bd89fdfc-kjdj9\" (UID: \"d804b100-c032-4251-8baf-c40f1e01f919\") " pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:01.379822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.379650 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af95fda-8dff-495e-8d59-334e7aeca7f8-cert\") pod \"odh-model-controller-769875db56-8lglj\" (UID: \"1af95fda-8dff-495e-8d59-334e7aeca7f8\") " pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.379822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.379681 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvlx\" (UniqueName: \"kubernetes.io/projected/1af95fda-8dff-495e-8d59-334e7aeca7f8-kube-api-access-6lvlx\") pod \"odh-model-controller-769875db56-8lglj\" (UID: \"1af95fda-8dff-495e-8d59-334e7aeca7f8\") " pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.379822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.379705 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af95fda-8dff-495e-8d59-334e7aeca7f8-metrics-certs\") pod \"odh-model-controller-769875db56-8lglj\" (UID: \"1af95fda-8dff-495e-8d59-334e7aeca7f8\") " pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.379822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.379730 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d804b100-c032-4251-8baf-c40f1e01f919-tls-certs\") pod \"model-serving-api-75bd89fdfc-kjdj9\" (UID: \"d804b100-c032-4251-8baf-c40f1e01f919\") " pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:01.382224 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.382191 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af95fda-8dff-495e-8d59-334e7aeca7f8-metrics-certs\") pod \"odh-model-controller-769875db56-8lglj\" (UID: \"1af95fda-8dff-495e-8d59-334e7aeca7f8\") " pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.382349 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.382231 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af95fda-8dff-495e-8d59-334e7aeca7f8-cert\") pod \"odh-model-controller-769875db56-8lglj\" (UID: \"1af95fda-8dff-495e-8d59-334e7aeca7f8\") " pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.382403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.382376 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d804b100-c032-4251-8baf-c40f1e01f919-tls-certs\") pod \"model-serving-api-75bd89fdfc-kjdj9\" (UID: \"d804b100-c032-4251-8baf-c40f1e01f919\") " pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:01.403650 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.403623 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45pb5\" (UniqueName: \"kubernetes.io/projected/d804b100-c032-4251-8baf-c40f1e01f919-kube-api-access-45pb5\") pod \"model-serving-api-75bd89fdfc-kjdj9\" (UID: \"d804b100-c032-4251-8baf-c40f1e01f919\") " pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:01.403800 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.403777 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvlx\" (UniqueName: \"kubernetes.io/projected/1af95fda-8dff-495e-8d59-334e7aeca7f8-kube-api-access-6lvlx\") pod \"odh-model-controller-769875db56-8lglj\" (UID: \"1af95fda-8dff-495e-8d59-334e7aeca7f8\") " pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.495954 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.495868 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:01.516210 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.516172 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:01.787405 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.787277 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-kjdj9"] Nov 03 10:36:01.789885 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:36:01.789859 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd804b100_c032_4251_8baf_c40f1e01f919.slice/crio-0b42e984aafde41543a7d25e43aadc77ee6dfa3a1970ff14a6de4aaf8722a7d5 WatchSource:0}: Error finding container 0b42e984aafde41543a7d25e43aadc77ee6dfa3a1970ff14a6de4aaf8722a7d5: Status 404 returned error can't find the container with id 0b42e984aafde41543a7d25e43aadc77ee6dfa3a1970ff14a6de4aaf8722a7d5 Nov 03 10:36:01.809062 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:01.809037 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8lglj"] Nov 03 10:36:01.811115 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:36:01.811080 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af95fda_8dff_495e_8d59_334e7aeca7f8.slice/crio-52a7bbdf23c281298abfe4ba08a45ea0a76681f877c228dae7f56abbf174cbf4 WatchSource:0}: Error finding container 52a7bbdf23c281298abfe4ba08a45ea0a76681f877c228dae7f56abbf174cbf4: Status 404 returned error can't find the container with id 52a7bbdf23c281298abfe4ba08a45ea0a76681f877c228dae7f56abbf174cbf4 Nov 03 10:36:02.376946 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:02.376908 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8lglj" event={"ID":"1af95fda-8dff-495e-8d59-334e7aeca7f8","Type":"ContainerStarted","Data":"52a7bbdf23c281298abfe4ba08a45ea0a76681f877c228dae7f56abbf174cbf4"} Nov 03 10:36:02.378104 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:02.378071 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" event={"ID":"d804b100-c032-4251-8baf-c40f1e01f919","Type":"ContainerStarted","Data":"0b42e984aafde41543a7d25e43aadc77ee6dfa3a1970ff14a6de4aaf8722a7d5"} Nov 03 10:36:04.392743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:04.392704 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" event={"ID":"d804b100-c032-4251-8baf-c40f1e01f919","Type":"ContainerStarted","Data":"34e3f451ec73ef9c7a49539fc5c41add927595eb66f6883cc116ea8397300d14"} Nov 03 10:36:04.393215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:04.392813 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:04.413004 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:04.412951 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" podStartSLOduration=1.229953052 podStartE2EDuration="3.412937738s" totalImagesPullingTime="2.182984686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:36:04.410473778 +0000 UTC m=+442.395345074" watchObservedRunningTime="2026-11-03 10:36:04.412937738 +0000 UTC m=+442.397809021" Nov 03 10:36:05.398821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:05.398782 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8lglj" event={"ID":"1af95fda-8dff-495e-8d59-334e7aeca7f8","Type":"ContainerStarted","Data":"1cb3f3ed575f2836aa38ef94a903132cb57313113c481aefc1df14125a03a28b"} Nov 03 10:36:05.399252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:05.399015 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:05.414722 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:05.414655 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-8lglj" podStartSLOduration=1.418366166 podStartE2EDuration="4.414642549s" totalImagesPullingTime="2.996276383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:36:05.414430757 +0000 UTC m=+443.399302067" watchObservedRunningTime="2026-11-03 10:36:05.414642549 +0000 UTC m=+443.399513865" Nov 03 10:36:15.403522 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:15.403491 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-kjdj9" Nov 03 10:36:16.405214 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:16.405181 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-8lglj" Nov 03 10:36:20.001100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.001066 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c"] Nov 03 10:36:20.005528 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.005500 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.012898 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.012870 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c"] Nov 03 10:36:20.014868 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.014841 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 10:36:20.014985 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.014909 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 10:36:20.014985 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.014981 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-dgjbt\"" Nov 03 10:36:20.015089 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.015036 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 10:36:20.050648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.050610 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad41c321-4063-400c-9415-2574d48dcc5b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.050648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.050645 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad41c321-4063-400c-9415-2574d48dcc5b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.050862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.050686 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ad41c321-4063-400c-9415-2574d48dcc5b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.050862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.050815 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv6bd\" (UniqueName: \"kubernetes.io/projected/ad41c321-4063-400c-9415-2574d48dcc5b-kube-api-access-qv6bd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.151732 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.151688 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad41c321-4063-400c-9415-2574d48dcc5b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.151732 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.151737 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad41c321-4063-400c-9415-2574d48dcc5b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.151963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.151777 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ad41c321-4063-400c-9415-2574d48dcc5b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.151963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.151813 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv6bd\" (UniqueName: \"kubernetes.io/projected/ad41c321-4063-400c-9415-2574d48dcc5b-kube-api-access-qv6bd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.151963 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:36:20.151867 5086 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 10:36:20.152093 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:36:20.151973 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad41c321-4063-400c-9415-2574d48dcc5b-tls-cert podName:ad41c321-4063-400c-9415-2574d48dcc5b nodeName:}" failed. No retries permitted until 2026-11-03 10:36:20.651952485 +0000 UTC m=+458.636823767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/ad41c321-4063-400c-9415-2574d48dcc5b-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" (UID: "ad41c321-4063-400c-9415-2574d48dcc5b") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 10:36:20.152630 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.152606 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad41c321-4063-400c-9415-2574d48dcc5b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.152862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.152847 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ad41c321-4063-400c-9415-2574d48dcc5b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.160172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.160147 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv6bd\" (UniqueName: \"kubernetes.io/projected/ad41c321-4063-400c-9415-2574d48dcc5b-kube-api-access-qv6bd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.655826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.655789 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad41c321-4063-400c-9415-2574d48dcc5b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.658172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.658142 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad41c321-4063-400c-9415-2574d48dcc5b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9pz5c\" (UID: \"ad41c321-4063-400c-9415-2574d48dcc5b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:20.923699 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:20.923603 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:21.197060 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:21.197025 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c"] Nov 03 10:36:21.199633 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:36:21.199605 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad41c321_4063_400c_9415_2574d48dcc5b.slice/crio-11ed40d0bc47a7bf71d5b77e373f56bf9a8bef31aaeac71ea71f0fad44bc6384 WatchSource:0}: Error finding container 11ed40d0bc47a7bf71d5b77e373f56bf9a8bef31aaeac71ea71f0fad44bc6384: Status 404 returned error can't find the container with id 11ed40d0bc47a7bf71d5b77e373f56bf9a8bef31aaeac71ea71f0fad44bc6384 Nov 03 10:36:21.475628 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:21.475544 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" event={"ID":"ad41c321-4063-400c-9415-2574d48dcc5b","Type":"ContainerStarted","Data":"11ed40d0bc47a7bf71d5b77e373f56bf9a8bef31aaeac71ea71f0fad44bc6384"} Nov 03 10:36:24.284909 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:36:24.284840 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:36:33.545431 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:33.545389 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" event={"ID":"ad41c321-4063-400c-9415-2574d48dcc5b","Type":"ContainerStarted","Data":"cbb17ab99a57af15e0813004b9a4619cc7d202b783845ce2f2949af817f60099"} Nov 03 10:36:33.546037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:33.545458 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:33.560623 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:33.560570 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" podStartSLOduration=3.060696806 podStartE2EDuration="14.560552366s" totalImagesPullingTime="11.49985556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:36:33.560027706 +0000 UTC m=+471.544898979" watchObservedRunningTime="2026-11-03 10:36:33.560552366 +0000 UTC m=+471.545423649" Nov 03 10:36:44.556496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:36:44.556463 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9pz5c" Nov 03 10:36:54.273808 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:36:54.273717 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:37:01.840506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:01.840464 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zmwhg"] Nov 03 10:37:01.941561 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:01.941506 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zmwhg"] Nov 03 10:37:01.941729 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:01.941594 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zmwhg"] Nov 03 10:37:01.941729 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:01.941616 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxx4q"] Nov 03 10:37:01.941729 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:01.941619 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zmwhg" Nov 03 10:37:01.942106 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:37:01.942078 5086 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sdclj], unattached volumes=[], failed to process volumes=[kube-api-access-sdclj]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-zmwhg" podUID="ad5f9f6e-872f-47f8-82bd-101a5d465653" Nov 03 10:37:01.975648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:01.975619 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxx4q"] Nov 03 10:37:01.975793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:01.975705 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:01.978893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:01.978872 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 10:37:02.026719 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.026677 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm79h\" (UniqueName: \"kubernetes.io/projected/372f53ac-4c7f-427c-9a3a-5b04fba64a10-kube-api-access-qm79h\") pod \"authorino-6889cbd8c4-rxx4q\" (UID: \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\") " pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:02.026900 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.026812 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdclj\" (UniqueName: \"kubernetes.io/projected/ad5f9f6e-872f-47f8-82bd-101a5d465653-kube-api-access-sdclj\") pod \"authorino-6f6944bcdd-zmwhg\" (UID: \"ad5f9f6e-872f-47f8-82bd-101a5d465653\") " pod="kuadrant-system/authorino-6f6944bcdd-zmwhg" Nov 03 10:37:02.026900 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.026883 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/372f53ac-4c7f-427c-9a3a-5b04fba64a10-tls-cert\") pod \"authorino-6889cbd8c4-rxx4q\" (UID: \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\") " pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:02.127619 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.127538 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdclj\" (UniqueName: \"kubernetes.io/projected/ad5f9f6e-872f-47f8-82bd-101a5d465653-kube-api-access-sdclj\") pod \"authorino-6f6944bcdd-zmwhg\" (UID: \"ad5f9f6e-872f-47f8-82bd-101a5d465653\") " pod="kuadrant-system/authorino-6f6944bcdd-zmwhg" Nov 03 10:37:02.127619 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.127577 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/372f53ac-4c7f-427c-9a3a-5b04fba64a10-tls-cert\") pod \"authorino-6889cbd8c4-rxx4q\" (UID: \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\") " pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:02.127833 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.127625 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm79h\" (UniqueName: \"kubernetes.io/projected/372f53ac-4c7f-427c-9a3a-5b04fba64a10-kube-api-access-qm79h\") pod \"authorino-6889cbd8c4-rxx4q\" (UID: \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\") " pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:02.130359 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.130326 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/372f53ac-4c7f-427c-9a3a-5b04fba64a10-tls-cert\") pod \"authorino-6889cbd8c4-rxx4q\" (UID: \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\") " pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:02.135614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.135584 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm79h\" (UniqueName: \"kubernetes.io/projected/372f53ac-4c7f-427c-9a3a-5b04fba64a10-kube-api-access-qm79h\") pod \"authorino-6889cbd8c4-rxx4q\" (UID: \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\") " pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:02.135614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.135604 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdclj\" (UniqueName: \"kubernetes.io/projected/ad5f9f6e-872f-47f8-82bd-101a5d465653-kube-api-access-sdclj\") pod \"authorino-6f6944bcdd-zmwhg\" (UID: \"ad5f9f6e-872f-47f8-82bd-101a5d465653\") " pod="kuadrant-system/authorino-6f6944bcdd-zmwhg" Nov 03 10:37:02.198046 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.198011 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxx4q"] Nov 03 10:37:02.198248 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.198236 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:02.220524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.220493 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c989b787c-nvj7f"] Nov 03 10:37:02.226166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.226141 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:37:02.232754 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.232727 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c989b787c-nvj7f"] Nov 03 10:37:02.329110 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.329070 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmrw\" (UniqueName: \"kubernetes.io/projected/00f207cd-eefa-430a-a7e9-36a30dc99f16-kube-api-access-tmmrw\") pod \"authorino-c989b787c-nvj7f\" (UID: \"00f207cd-eefa-430a-a7e9-36a30dc99f16\") " pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:37:02.329277 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.329253 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f207cd-eefa-430a-a7e9-36a30dc99f16-tls-cert\") pod \"authorino-c989b787c-nvj7f\" (UID: \"00f207cd-eefa-430a-a7e9-36a30dc99f16\") " pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:37:02.430113 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.430025 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f207cd-eefa-430a-a7e9-36a30dc99f16-tls-cert\") pod \"authorino-c989b787c-nvj7f\" (UID: \"00f207cd-eefa-430a-a7e9-36a30dc99f16\") " pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:37:02.430113 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.430084 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmrw\" (UniqueName: \"kubernetes.io/projected/00f207cd-eefa-430a-a7e9-36a30dc99f16-kube-api-access-tmmrw\") pod \"authorino-c989b787c-nvj7f\" (UID: \"00f207cd-eefa-430a-a7e9-36a30dc99f16\") " pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:37:02.432663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.432637 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f207cd-eefa-430a-a7e9-36a30dc99f16-tls-cert\") pod \"authorino-c989b787c-nvj7f\" (UID: \"00f207cd-eefa-430a-a7e9-36a30dc99f16\") " pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:37:02.438198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.438169 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmrw\" (UniqueName: \"kubernetes.io/projected/00f207cd-eefa-430a-a7e9-36a30dc99f16-kube-api-access-tmmrw\") pod \"authorino-c989b787c-nvj7f\" (UID: \"00f207cd-eefa-430a-a7e9-36a30dc99f16\") " pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:37:02.487989 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.487960 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxx4q"] Nov 03 10:37:02.490112 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:37:02.490087 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod372f53ac_4c7f_427c_9a3a_5b04fba64a10.slice/crio-ef208a1694672ccc2556b44d67f979c999f11d03256a3a8855d3db62a62ff964 WatchSource:0}: Error finding container ef208a1694672ccc2556b44d67f979c999f11d03256a3a8855d3db62a62ff964: Status 404 returned error can't find the container with id ef208a1694672ccc2556b44d67f979c999f11d03256a3a8855d3db62a62ff964 Nov 03 10:37:02.578368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.578331 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:37:02.685716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.685638 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" event={"ID":"372f53ac-4c7f-427c-9a3a-5b04fba64a10","Type":"ContainerStarted","Data":"ef208a1694672ccc2556b44d67f979c999f11d03256a3a8855d3db62a62ff964"} Nov 03 10:37:02.685716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.685704 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zmwhg" Nov 03 10:37:02.717876 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.717843 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zmwhg" Nov 03 10:37:02.834517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.834479 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5f9f6e-872f-47f8-82bd-101a5d465653-kube-api-access-sdclj\" (UniqueName: \"kubernetes.io/projected/ad5f9f6e-872f-47f8-82bd-101a5d465653-kube-api-access-sdclj\") pod \"ad5f9f6e-872f-47f8-82bd-101a5d465653\" (UID: \"ad5f9f6e-872f-47f8-82bd-101a5d465653\") " Nov 03 10:37:02.836787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.836754 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5f9f6e-872f-47f8-82bd-101a5d465653-kube-api-access-sdclj" pod "ad5f9f6e-872f-47f8-82bd-101a5d465653" (UID: "ad5f9f6e-872f-47f8-82bd-101a5d465653"). InnerVolumeSpecName "kube-api-access-sdclj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:37:02.869086 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.869061 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c989b787c-nvj7f"] Nov 03 10:37:02.871193 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:37:02.871170 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00f207cd_eefa_430a_a7e9_36a30dc99f16.slice/crio-a01c285fc6eb0ff34a08d728ae9cec39e1c428fd150b2dd814918137bdfb7352 WatchSource:0}: Error finding container a01c285fc6eb0ff34a08d728ae9cec39e1c428fd150b2dd814918137bdfb7352: Status 404 returned error can't find the container with id a01c285fc6eb0ff34a08d728ae9cec39e1c428fd150b2dd814918137bdfb7352 Nov 03 10:37:02.935766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:02.935684 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdclj\" (UniqueName: \"kubernetes.io/projected/ad5f9f6e-872f-47f8-82bd-101a5d465653-kube-api-access-sdclj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:37:03.692387 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:03.692333 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" event={"ID":"372f53ac-4c7f-427c-9a3a-5b04fba64a10","Type":"ContainerStarted","Data":"026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2"} Nov 03 10:37:03.692588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:03.692391 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" podUID="372f53ac-4c7f-427c-9a3a-5b04fba64a10" containerName="authorino" containerID="cri-o://026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2" gracePeriod=30 Nov 03 10:37:03.694075 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:03.694048 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c989b787c-nvj7f" event={"ID":"00f207cd-eefa-430a-a7e9-36a30dc99f16","Type":"ContainerStarted","Data":"a01c285fc6eb0ff34a08d728ae9cec39e1c428fd150b2dd814918137bdfb7352"} Nov 03 10:37:03.694183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:03.694110 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zmwhg" Nov 03 10:37:03.708691 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:03.708642 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" podStartSLOduration=1.677393977 podStartE2EDuration="2.708628702s" totalImagesPullingTime="1.031234725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:37:03.707237833 +0000 UTC m=+501.692109117" watchObservedRunningTime="2026-11-03 10:37:03.708628702 +0000 UTC m=+501.693499963" Nov 03 10:37:03.731016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:03.730984 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zmwhg"] Nov 03 10:37:03.736859 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:03.736833 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zmwhg"] Nov 03 10:37:03.981701 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:03.981677 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:04.044227 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.044193 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/372f53ac-4c7f-427c-9a3a-5b04fba64a10-tls-cert\" (UniqueName: \"kubernetes.io/secret/372f53ac-4c7f-427c-9a3a-5b04fba64a10-tls-cert\") pod \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\" (UID: \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\") " Nov 03 10:37:04.044401 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.044353 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372f53ac-4c7f-427c-9a3a-5b04fba64a10-kube-api-access-qm79h\" (UniqueName: \"kubernetes.io/projected/372f53ac-4c7f-427c-9a3a-5b04fba64a10-kube-api-access-qm79h\") pod \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\" (UID: \"372f53ac-4c7f-427c-9a3a-5b04fba64a10\") " Nov 03 10:37:04.046867 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.046825 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372f53ac-4c7f-427c-9a3a-5b04fba64a10-kube-api-access-qm79h" pod "372f53ac-4c7f-427c-9a3a-5b04fba64a10" (UID: "372f53ac-4c7f-427c-9a3a-5b04fba64a10"). InnerVolumeSpecName "kube-api-access-qm79h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:37:04.056508 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.056476 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372f53ac-4c7f-427c-9a3a-5b04fba64a10-tls-cert" pod "372f53ac-4c7f-427c-9a3a-5b04fba64a10" (UID: "372f53ac-4c7f-427c-9a3a-5b04fba64a10"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:37:04.145655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.145622 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm79h\" (UniqueName: \"kubernetes.io/projected/372f53ac-4c7f-427c-9a3a-5b04fba64a10-kube-api-access-qm79h\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:37:04.145773 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.145659 5086 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/372f53ac-4c7f-427c-9a3a-5b04fba64a10-tls-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:37:04.186500 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.186464 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5f9f6e-872f-47f8-82bd-101a5d465653" path="/var/lib/kubelet/pods/ad5f9f6e-872f-47f8-82bd-101a5d465653/volumes" Nov 03 10:37:04.700601 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.700562 5086 generic.go:356] "Generic (PLEG): container finished" podID="372f53ac-4c7f-427c-9a3a-5b04fba64a10" containerID="026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2" exitCode=0 Nov 03 10:37:04.700601 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.700603 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" Nov 03 10:37:04.700851 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.700644 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" event={"ID":"372f53ac-4c7f-427c-9a3a-5b04fba64a10","Type":"ContainerDied","Data":"026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2"} Nov 03 10:37:04.700851 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.700688 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-rxx4q" event={"ID":"372f53ac-4c7f-427c-9a3a-5b04fba64a10","Type":"ContainerDied","Data":"ef208a1694672ccc2556b44d67f979c999f11d03256a3a8855d3db62a62ff964"} Nov 03 10:37:04.700851 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.700711 5086 scope.go:122] "RemoveContainer" containerID="026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2" Nov 03 10:37:04.702596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.702550 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c989b787c-nvj7f" event={"ID":"00f207cd-eefa-430a-a7e9-36a30dc99f16","Type":"ContainerStarted","Data":"8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b"} Nov 03 10:37:04.709804 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.709787 5086 scope.go:122] "RemoveContainer" containerID="026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2" Nov 03 10:37:04.710078 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:37:04.710056 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2\": container with ID starting with 026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2 not found: ID does not exist" containerID="026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2" Nov 03 10:37:04.710159 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.710086 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2"} err="failed to get container status \"026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2\": rpc error: code = NotFound desc = could not find container \"026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2\": container with ID starting with 026e84f4067337578ccedc1d522e0b1f2ea199cc273862860966878c02d49ec2 not found: ID does not exist" Nov 03 10:37:04.721013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.720968 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c989b787c-nvj7f" podStartSLOduration=1.728982681 podStartE2EDuration="2.720954967s" totalImagesPullingTime="991.972286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:37:04.71889104 +0000 UTC m=+502.703762327" watchObservedRunningTime="2026-11-03 10:37:04.720954967 +0000 UTC m=+502.705826250" Nov 03 10:37:04.735102 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.733358 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxx4q"] Nov 03 10:37:04.735102 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.734971 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxx4q"] Nov 03 10:37:04.738473 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.738450 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lcg5t"] Nov 03 10:37:04.738646 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.738626 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" podUID="8e45a9db-16bd-4e4b-87d1-e09926df4a75" containerName="authorino" containerID="cri-o://989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9" gracePeriod=30 Nov 03 10:37:04.998877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:04.998851 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" Nov 03 10:37:05.055037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.055008 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e45a9db-16bd-4e4b-87d1-e09926df4a75-kube-api-access-zmdjv\" (UniqueName: \"kubernetes.io/projected/8e45a9db-16bd-4e4b-87d1-e09926df4a75-kube-api-access-zmdjv\") pod \"8e45a9db-16bd-4e4b-87d1-e09926df4a75\" (UID: \"8e45a9db-16bd-4e4b-87d1-e09926df4a75\") " Nov 03 10:37:05.057148 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.057116 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e45a9db-16bd-4e4b-87d1-e09926df4a75-kube-api-access-zmdjv" pod "8e45a9db-16bd-4e4b-87d1-e09926df4a75" (UID: "8e45a9db-16bd-4e4b-87d1-e09926df4a75"). InnerVolumeSpecName "kube-api-access-zmdjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:37:05.156047 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.156005 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmdjv\" (UniqueName: \"kubernetes.io/projected/8e45a9db-16bd-4e4b-87d1-e09926df4a75-kube-api-access-zmdjv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:37:05.710027 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.709989 5086 generic.go:356] "Generic (PLEG): container finished" podID="8e45a9db-16bd-4e4b-87d1-e09926df4a75" containerID="989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9" exitCode=0 Nov 03 10:37:05.710237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.710032 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" Nov 03 10:37:05.710237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.710069 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" event={"ID":"8e45a9db-16bd-4e4b-87d1-e09926df4a75","Type":"ContainerDied","Data":"989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9"} Nov 03 10:37:05.710237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.710104 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lcg5t" event={"ID":"8e45a9db-16bd-4e4b-87d1-e09926df4a75","Type":"ContainerDied","Data":"c4b88572cdbe0bcb8e7b1d3a09043b20a0765c0bbdb91e9732bf94c24455fb57"} Nov 03 10:37:05.710237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.710123 5086 scope.go:122] "RemoveContainer" containerID="989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9" Nov 03 10:37:05.719548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.719533 5086 scope.go:122] "RemoveContainer" containerID="989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9" Nov 03 10:37:05.719794 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:37:05.719776 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9\": container with ID starting with 989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9 not found: ID does not exist" containerID="989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9" Nov 03 10:37:05.719856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.719801 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9"} err="failed to get container status \"989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9\": rpc error: code = NotFound desc = could not find container \"989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9\": container with ID starting with 989bc8c0f19f5caa562188531abb094c8dc4767c9b66e67b4d0fac6cf487b7e9 not found: ID does not exist" Nov 03 10:37:05.729695 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.729674 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lcg5t"] Nov 03 10:37:05.733258 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:05.733238 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lcg5t"] Nov 03 10:37:06.169047 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:06.169011 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372f53ac-4c7f-427c-9a3a-5b04fba64a10" path="/var/lib/kubelet/pods/372f53ac-4c7f-427c-9a3a-5b04fba64a10/volumes" Nov 03 10:37:06.169442 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:06.169373 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e45a9db-16bd-4e4b-87d1-e09926df4a75" path="/var/lib/kubelet/pods/8e45a9db-16bd-4e4b-87d1-e09926df4a75/volumes" Nov 03 10:37:24.273110 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:37:24.273074 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:37:50.411168 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.411132 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2ts72"] Nov 03 10:37:50.411609 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.411549 5086 cpu_manager.go:414] "containerMap: removing container" podUID="8e45a9db-16bd-4e4b-87d1-e09926df4a75" containerName="authorino" Nov 03 10:37:50.411609 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.411560 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e45a9db-16bd-4e4b-87d1-e09926df4a75" containerName="authorino" Nov 03 10:37:50.411609 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.411581 5086 cpu_manager.go:414] "containerMap: removing container" podUID="372f53ac-4c7f-427c-9a3a-5b04fba64a10" containerName="authorino" Nov 03 10:37:50.411609 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.411587 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372f53ac-4c7f-427c-9a3a-5b04fba64a10" containerName="authorino" Nov 03 10:37:50.411785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.411644 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e45a9db-16bd-4e4b-87d1-e09926df4a75" containerName="authorino" Nov 03 10:37:50.411785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.411655 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="372f53ac-4c7f-427c-9a3a-5b04fba64a10" containerName="authorino" Nov 03 10:37:50.416225 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.416206 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:37:50.421502 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.421440 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2ts72"] Nov 03 10:37:50.423088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.423067 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 10:37:50.423223 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.423113 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 10:37:50.423322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.423137 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-p29fg\"" Nov 03 10:37:50.448250 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.448220 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5064a44e-0788-478f-a12e-c52e14a64722-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-2ts72\" (UID: \"5064a44e-0788-478f-a12e-c52e14a64722\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:37:50.448386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.448273 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxw2r\" (UniqueName: \"kubernetes.io/projected/5064a44e-0788-478f-a12e-c52e14a64722-kube-api-access-qxw2r\") pod \"ai-gateway-controller-5d8dddb676-2ts72\" (UID: \"5064a44e-0788-478f-a12e-c52e14a64722\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:37:50.549663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.549625 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5064a44e-0788-478f-a12e-c52e14a64722-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-2ts72\" (UID: \"5064a44e-0788-478f-a12e-c52e14a64722\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:37:50.549663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.549673 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxw2r\" (UniqueName: \"kubernetes.io/projected/5064a44e-0788-478f-a12e-c52e14a64722-kube-api-access-qxw2r\") pod \"ai-gateway-controller-5d8dddb676-2ts72\" (UID: \"5064a44e-0788-478f-a12e-c52e14a64722\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:37:50.552198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.552170 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5064a44e-0788-478f-a12e-c52e14a64722-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-2ts72\" (UID: \"5064a44e-0788-478f-a12e-c52e14a64722\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:37:50.557273 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.557246 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxw2r\" (UniqueName: \"kubernetes.io/projected/5064a44e-0788-478f-a12e-c52e14a64722-kube-api-access-qxw2r\") pod \"ai-gateway-controller-5d8dddb676-2ts72\" (UID: \"5064a44e-0788-478f-a12e-c52e14a64722\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:37:50.735353 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:50.735244 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:37:51.005999 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:51.005919 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2ts72"] Nov 03 10:37:51.009221 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:37:51.009194 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5064a44e_0788_478f_a12e_c52e14a64722.slice/crio-93512e63c6159fbbc80a75c346865825f4939b63027991f614480a1a88aea591 WatchSource:0}: Error finding container 93512e63c6159fbbc80a75c346865825f4939b63027991f614480a1a88aea591: Status 404 returned error can't find the container with id 93512e63c6159fbbc80a75c346865825f4939b63027991f614480a1a88aea591 Nov 03 10:37:51.936847 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:51.936809 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" event={"ID":"5064a44e-0788-478f-a12e-c52e14a64722","Type":"ContainerStarted","Data":"93512e63c6159fbbc80a75c346865825f4939b63027991f614480a1a88aea591"} Nov 03 10:37:54.281031 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:37:54.280989 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:37:54.953063 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:54.952974 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" event={"ID":"5064a44e-0788-478f-a12e-c52e14a64722","Type":"ContainerStarted","Data":"d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49"} Nov 03 10:37:54.953216 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:54.953118 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:37:54.969203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:37:54.969149 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" podStartSLOduration=1.297777327 podStartE2EDuration="4.969137872s" totalImagesPullingTime="3.671360545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:37:54.967012481 +0000 UTC m=+552.951883766" watchObservedRunningTime="2026-11-03 10:37:54.969137872 +0000 UTC m=+552.954009155" Nov 03 10:38:05.962361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:05.962327 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:38:06.196330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.196011 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6b675dd85b-h27pj"] Nov 03 10:38:06.201227 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.201201 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:06.207188 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.207162 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6b675dd85b-h27pj"] Nov 03 10:38:06.284178 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.284078 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8dm\" (UniqueName: \"kubernetes.io/projected/be5179f0-1851-42f5-8429-8ffa5892af05-kube-api-access-fm8dm\") pod \"ai-gateway-controller-6b675dd85b-h27pj\" (UID: \"be5179f0-1851-42f5-8429-8ffa5892af05\") " pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:06.284376 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.284252 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be5179f0-1851-42f5-8429-8ffa5892af05-metrics-certs\") pod \"ai-gateway-controller-6b675dd85b-h27pj\" (UID: \"be5179f0-1851-42f5-8429-8ffa5892af05\") " pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:06.385485 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.385453 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be5179f0-1851-42f5-8429-8ffa5892af05-metrics-certs\") pod \"ai-gateway-controller-6b675dd85b-h27pj\" (UID: \"be5179f0-1851-42f5-8429-8ffa5892af05\") " pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:06.385640 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.385509 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8dm\" (UniqueName: \"kubernetes.io/projected/be5179f0-1851-42f5-8429-8ffa5892af05-kube-api-access-fm8dm\") pod \"ai-gateway-controller-6b675dd85b-h27pj\" (UID: \"be5179f0-1851-42f5-8429-8ffa5892af05\") " pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:06.387949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.387911 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be5179f0-1851-42f5-8429-8ffa5892af05-metrics-certs\") pod \"ai-gateway-controller-6b675dd85b-h27pj\" (UID: \"be5179f0-1851-42f5-8429-8ffa5892af05\") " pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:06.393395 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.393372 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8dm\" (UniqueName: \"kubernetes.io/projected/be5179f0-1851-42f5-8429-8ffa5892af05-kube-api-access-fm8dm\") pod \"ai-gateway-controller-6b675dd85b-h27pj\" (UID: \"be5179f0-1851-42f5-8429-8ffa5892af05\") " pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:06.516608 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.516574 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:06.786700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:06.786675 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6b675dd85b-h27pj"] Nov 03 10:38:06.788774 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:38:06.788745 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5179f0_1851_42f5_8429_8ffa5892af05.slice/crio-c61d81d837c575ac4ac083cca36875d286a755a13c44a2550287dc5859bbb3c3 WatchSource:0}: Error finding container c61d81d837c575ac4ac083cca36875d286a755a13c44a2550287dc5859bbb3c3: Status 404 returned error can't find the container with id c61d81d837c575ac4ac083cca36875d286a755a13c44a2550287dc5859bbb3c3 Nov 03 10:38:07.021255 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:07.021218 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" event={"ID":"be5179f0-1851-42f5-8429-8ffa5892af05","Type":"ContainerStarted","Data":"c61d81d837c575ac4ac083cca36875d286a755a13c44a2550287dc5859bbb3c3"} Nov 03 10:38:08.027752 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:08.027719 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" event={"ID":"be5179f0-1851-42f5-8429-8ffa5892af05","Type":"ContainerStarted","Data":"e2dcebb5b0e8eaa147b619795d335d179d3f44c2c345247fa75e723abb0b6efd"} Nov 03 10:38:08.028144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:08.027782 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:08.044487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:08.044439 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" podStartSLOduration=1.611107676 podStartE2EDuration="2.044425405s" totalImagesPullingTime="433.317729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:08.042725507 +0000 UTC m=+566.027596820" watchObservedRunningTime="2026-11-03 10:38:08.044425405 +0000 UTC m=+566.029296687" Nov 03 10:38:19.035682 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.035645 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6b675dd85b-h27pj" Nov 03 10:38:19.072210 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.072178 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2ts72"] Nov 03 10:38:19.072406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.072384 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" podUID="5064a44e-0788-478f-a12e-c52e14a64722" containerName="manager" containerID="cri-o://d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49" gracePeriod=30 Nov 03 10:38:19.318179 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.318151 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:38:19.398316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.398261 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5064a44e-0788-478f-a12e-c52e14a64722-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5064a44e-0788-478f-a12e-c52e14a64722-metrics-certs\") pod \"5064a44e-0788-478f-a12e-c52e14a64722\" (UID: \"5064a44e-0788-478f-a12e-c52e14a64722\") " Nov 03 10:38:19.398519 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.398349 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5064a44e-0788-478f-a12e-c52e14a64722-kube-api-access-qxw2r\" (UniqueName: \"kubernetes.io/projected/5064a44e-0788-478f-a12e-c52e14a64722-kube-api-access-qxw2r\") pod \"5064a44e-0788-478f-a12e-c52e14a64722\" (UID: \"5064a44e-0788-478f-a12e-c52e14a64722\") " Nov 03 10:38:19.400592 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.400562 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5064a44e-0788-478f-a12e-c52e14a64722-kube-api-access-qxw2r" pod "5064a44e-0788-478f-a12e-c52e14a64722" (UID: "5064a44e-0788-478f-a12e-c52e14a64722"). InnerVolumeSpecName "kube-api-access-qxw2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:38:19.400706 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.400622 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5064a44e-0788-478f-a12e-c52e14a64722-metrics-certs" pod "5064a44e-0788-478f-a12e-c52e14a64722" (UID: "5064a44e-0788-478f-a12e-c52e14a64722"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:38:19.499312 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.499270 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5064a44e-0788-478f-a12e-c52e14a64722-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:38:19.499481 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:19.499335 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxw2r\" (UniqueName: \"kubernetes.io/projected/5064a44e-0788-478f-a12e-c52e14a64722-kube-api-access-qxw2r\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:38:20.092282 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.092247 5086 generic.go:356] "Generic (PLEG): container finished" podID="5064a44e-0788-478f-a12e-c52e14a64722" containerID="d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49" exitCode=0 Nov 03 10:38:20.092733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.092309 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" Nov 03 10:38:20.092733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.092315 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" event={"ID":"5064a44e-0788-478f-a12e-c52e14a64722","Type":"ContainerDied","Data":"d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49"} Nov 03 10:38:20.092733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.092386 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-2ts72" event={"ID":"5064a44e-0788-478f-a12e-c52e14a64722","Type":"ContainerDied","Data":"93512e63c6159fbbc80a75c346865825f4939b63027991f614480a1a88aea591"} Nov 03 10:38:20.092733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.092407 5086 scope.go:122] "RemoveContainer" containerID="d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49" Nov 03 10:38:20.102908 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.102889 5086 scope.go:122] "RemoveContainer" containerID="d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49" Nov 03 10:38:20.103194 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:38:20.103172 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49\": container with ID starting with d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49 not found: ID does not exist" containerID="d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49" Nov 03 10:38:20.103231 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.103204 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49"} err="failed to get container status \"d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49\": rpc error: code = NotFound desc = could not find container \"d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49\": container with ID starting with d45abd8311b32f5c761de89d718718025ab8ca2d59d99be59cd95f0c09339e49 not found: ID does not exist" Nov 03 10:38:20.115367 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.115343 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2ts72"] Nov 03 10:38:20.117328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.117307 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2ts72"] Nov 03 10:38:20.169199 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:20.169167 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5064a44e-0788-478f-a12e-c52e14a64722" path="/var/lib/kubelet/pods/5064a44e-0788-478f-a12e-c52e14a64722/volumes" Nov 03 10:38:24.277820 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:38:24.277721 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:38:25.839110 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.839075 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5"] Nov 03 10:38:25.839703 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.839681 5086 cpu_manager.go:414] "containerMap: removing container" podUID="5064a44e-0788-478f-a12e-c52e14a64722" containerName="manager" Nov 03 10:38:25.839803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.839704 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5064a44e-0788-478f-a12e-c52e14a64722" containerName="manager" Nov 03 10:38:25.839870 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.839855 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="5064a44e-0788-478f-a12e-c52e14a64722" containerName="manager" Nov 03 10:38:25.842937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.842914 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:25.847158 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.847133 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 10:38:25.848127 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.848108 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 10:38:25.848208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.848131 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 10:38:25.848208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.848136 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4cvtj\"" Nov 03 10:38:25.852432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.852412 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5"] Nov 03 10:38:25.954537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.954502 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:25.954744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.954553 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:25.954744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.954669 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb46a330-9188-4da0-9d44-b2795c71995e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:25.954868 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.954748 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:25.954868 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.954780 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv62t\" (UniqueName: \"kubernetes.io/projected/bb46a330-9188-4da0-9d44-b2795c71995e-kube-api-access-vv62t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:25.954868 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:25.954859 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.056139 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.056100 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv62t\" (UniqueName: \"kubernetes.io/projected/bb46a330-9188-4da0-9d44-b2795c71995e-kube-api-access-vv62t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.056139 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.056142 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.056454 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.056320 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.056454 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.056398 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.056454 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.056441 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb46a330-9188-4da0-9d44-b2795c71995e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.056645 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.056482 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.056645 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.056629 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.056761 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.056739 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.056803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.056788 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.058459 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.058441 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb46a330-9188-4da0-9d44-b2795c71995e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.058803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.058782 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb46a330-9188-4da0-9d44-b2795c71995e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.063593 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.063573 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv62t\" (UniqueName: \"kubernetes.io/projected/bb46a330-9188-4da0-9d44-b2795c71995e-kube-api-access-vv62t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5\" (UID: \"bb46a330-9188-4da0-9d44-b2795c71995e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.157129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.157031 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:26.444276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:26.444251 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5"] Nov 03 10:38:26.447385 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:38:26.447357 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb46a330_9188_4da0_9d44_b2795c71995e.slice/crio-3bc780b671dd7b166e505eb299e7255a1f507140cba1e56ec6ef8342c77662af WatchSource:0}: Error finding container 3bc780b671dd7b166e505eb299e7255a1f507140cba1e56ec6ef8342c77662af: Status 404 returned error can't find the container with id 3bc780b671dd7b166e505eb299e7255a1f507140cba1e56ec6ef8342c77662af Nov 03 10:38:27.127638 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:27.127600 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" event={"ID":"bb46a330-9188-4da0-9d44-b2795c71995e","Type":"ContainerStarted","Data":"3bc780b671dd7b166e505eb299e7255a1f507140cba1e56ec6ef8342c77662af"} Nov 03 10:38:28.133813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:28.133775 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" event={"ID":"bb46a330-9188-4da0-9d44-b2795c71995e","Type":"ContainerStarted","Data":"a2326bcb5aa4652b4a1787c2bed82c45bd0c6450f0ff6b6227198057683ea74a"} Nov 03 10:38:28.149235 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:28.149183 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" podStartSLOduration=1.656947022 podStartE2EDuration="3.149169936s" totalImagesPullingTime="1.492222914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:28.148698511 +0000 UTC m=+586.133569810" watchObservedRunningTime="2026-11-03 10:38:28.149169936 +0000 UTC m=+586.134041219" Nov 03 10:38:28.157959 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:28.157937 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:29.887117 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.887084 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97"] Nov 03 10:38:29.891095 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.891066 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:29.900829 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.900807 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 10:38:29.901478 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.901447 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97"] Nov 03 10:38:29.992030 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.991993 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/266331b8-afce-492e-830c-933449e1a765-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:29.992030 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.992034 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:29.992270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.992058 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:29.992270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.992134 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwt9\" (UniqueName: \"kubernetes.io/projected/266331b8-afce-492e-830c-933449e1a765-kube-api-access-vkwt9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:29.992270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.992172 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:29.992270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:29.992225 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.092795 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.092760 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/266331b8-afce-492e-830c-933449e1a765-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.092956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.092842 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.092956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.092874 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.092956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.092900 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwt9\" (UniqueName: \"kubernetes.io/projected/266331b8-afce-492e-830c-933449e1a765-kube-api-access-vkwt9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.092956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.092932 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.093167 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.093011 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.093373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.093343 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.093373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.093367 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.093604 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.093580 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.095690 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.095667 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/266331b8-afce-492e-830c-933449e1a765-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.095979 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.095959 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/266331b8-afce-492e-830c-933449e1a765-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.101574 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.101547 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwt9\" (UniqueName: \"kubernetes.io/projected/266331b8-afce-492e-830c-933449e1a765-kube-api-access-vkwt9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-bpt97\" (UID: \"266331b8-afce-492e-830c-933449e1a765\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.208092 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.207987 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:30.484537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:30.484503 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97"] Nov 03 10:38:30.485841 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:38:30.485816 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266331b8_afce_492e_830c_933449e1a765.slice/crio-5965fa0dace9fdd5ec3c62dd23040c35d2528462cae816f190aba9eec8c73903 WatchSource:0}: Error finding container 5965fa0dace9fdd5ec3c62dd23040c35d2528462cae816f190aba9eec8c73903: Status 404 returned error can't find the container with id 5965fa0dace9fdd5ec3c62dd23040c35d2528462cae816f190aba9eec8c73903 Nov 03 10:38:31.149445 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:31.149402 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" event={"ID":"266331b8-afce-492e-830c-933449e1a765","Type":"ContainerStarted","Data":"00eaddd7ed11d63eedf3cb8822e1a55c809a6cbcb1133800298bcb739ac50380"} Nov 03 10:38:31.149894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:31.149487 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" event={"ID":"266331b8-afce-492e-830c-933449e1a765","Type":"ContainerStarted","Data":"5965fa0dace9fdd5ec3c62dd23040c35d2528462cae816f190aba9eec8c73903"} Nov 03 10:38:31.165639 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:31.165585 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" podStartSLOduration=1.863873038 podStartE2EDuration="2.165570897s" totalImagesPullingTime="301.697859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:31.165223555 +0000 UTC m=+589.150094839" watchObservedRunningTime="2026-11-03 10:38:31.165570897 +0000 UTC m=+589.150442177" Nov 03 10:38:31.208736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:31.208705 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:31.940809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:31.940775 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4"] Nov 03 10:38:31.945981 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:31.945956 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:31.954337 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:31.954097 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4"] Nov 03 10:38:31.954337 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:31.954334 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 10:38:32.011811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.011770 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.012164 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.012142 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsldv\" (UniqueName: \"kubernetes.io/projected/70328620-3e3d-4bd3-a573-80e8117c8d64-kube-api-access-xsldv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.012364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.012336 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.012496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.012374 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.012496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.012478 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.012639 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.012502 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70328620-3e3d-4bd3-a573-80e8117c8d64-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.113086 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.113044 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.113086 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.113084 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70328620-3e3d-4bd3-a573-80e8117c8d64-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.113335 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.113128 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.113335 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.113173 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsldv\" (UniqueName: \"kubernetes.io/projected/70328620-3e3d-4bd3-a573-80e8117c8d64-kube-api-access-xsldv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.113335 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.113287 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.113504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.113364 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.113589 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.113568 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.113653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.113636 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.113710 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.113652 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.115639 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.115612 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70328620-3e3d-4bd3-a573-80e8117c8d64-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.115833 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.115816 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70328620-3e3d-4bd3-a573-80e8117c8d64-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.124230 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.124202 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsldv\" (UniqueName: \"kubernetes.io/projected/70328620-3e3d-4bd3-a573-80e8117c8d64-kube-api-access-xsldv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4\" (UID: \"70328620-3e3d-4bd3-a573-80e8117c8d64\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.267384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.267244 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:32.569700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:32.569669 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4"] Nov 03 10:38:32.571396 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:38:32.571358 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70328620_3e3d_4bd3_a573_80e8117c8d64.slice/crio-130c808c17d7d819e88e1655c12b5db00a5e8c1e22ef234a0a533e0dc46486bb WatchSource:0}: Error finding container 130c808c17d7d819e88e1655c12b5db00a5e8c1e22ef234a0a533e0dc46486bb: Status 404 returned error can't find the container with id 130c808c17d7d819e88e1655c12b5db00a5e8c1e22ef234a0a533e0dc46486bb Nov 03 10:38:33.160657 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.160560 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" event={"ID":"70328620-3e3d-4bd3-a573-80e8117c8d64","Type":"ContainerStarted","Data":"9fb7263a38a298739dbd0196c909a102061760483e883a0ebe9e132a297eee98"} Nov 03 10:38:33.160657 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.160614 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" event={"ID":"70328620-3e3d-4bd3-a573-80e8117c8d64","Type":"ContainerStarted","Data":"130c808c17d7d819e88e1655c12b5db00a5e8c1e22ef234a0a533e0dc46486bb"} Nov 03 10:38:33.178618 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.178569 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" podStartSLOduration=1.8761883990000001 podStartE2EDuration="2.178556033s" totalImagesPullingTime="302.367634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:33.176793269 +0000 UTC m=+591.161664644" watchObservedRunningTime="2026-11-03 10:38:33.178556033 +0000 UTC m=+591.163427337" Nov 03 10:38:33.267902 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.267864 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:33.785428 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.785392 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r"] Nov 03 10:38:33.789760 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.789737 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:33.796488 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.796452 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 10:38:33.800740 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.800712 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r"] Nov 03 10:38:33.931315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.930751 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:33.931315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.930841 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:33.931315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.930878 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:33.931315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.930911 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:33.931315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.930937 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjplx\" (UniqueName: \"kubernetes.io/projected/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-kube-api-access-wjplx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:33.931315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:33.930974 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.031470 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.031436 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.031648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.031478 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.031648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.031506 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjplx\" (UniqueName: \"kubernetes.io/projected/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-kube-api-access-wjplx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.031648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.031540 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.031648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.031626 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.031858 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.031690 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.031989 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.031960 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.031989 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.031972 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.032198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.032069 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.033831 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.033809 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.034205 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.034186 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.039402 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.039339 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjplx\" (UniqueName: \"kubernetes.io/projected/a5af7ba9-03fa-4655-b7bd-3f18d457be7b-kube-api-access-wjplx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r\" (UID: \"a5af7ba9-03fa-4655-b7bd-3f18d457be7b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.107431 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.107393 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:34.400474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:34.400448 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r"] Nov 03 10:38:34.403314 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:38:34.403265 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5af7ba9_03fa_4655_b7bd_3f18d457be7b.slice/crio-70c5297a8f67daed01b7fd3eb21702c92584953638baf9c7d39beab4c58ca041 WatchSource:0}: Error finding container 70c5297a8f67daed01b7fd3eb21702c92584953638baf9c7d39beab4c58ca041: Status 404 returned error can't find the container with id 70c5297a8f67daed01b7fd3eb21702c92584953638baf9c7d39beab4c58ca041 Nov 03 10:38:35.175375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:35.175331 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" event={"ID":"a5af7ba9-03fa-4655-b7bd-3f18d457be7b","Type":"ContainerStarted","Data":"bf0c7ec03687ce61c75ed7feefb03311e0d20edeeb1eabe976871a76847ad495"} Nov 03 10:38:35.175556 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:35.175435 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" event={"ID":"a5af7ba9-03fa-4655-b7bd-3f18d457be7b","Type":"ContainerStarted","Data":"70c5297a8f67daed01b7fd3eb21702c92584953638baf9c7d39beab4c58ca041"} Nov 03 10:38:35.193393 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:35.193340 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" podStartSLOduration=1.903707995 podStartE2EDuration="2.193327336s" totalImagesPullingTime="289.619341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:35.190661374 +0000 UTC m=+593.175532669" watchObservedRunningTime="2026-11-03 10:38:35.193327336 +0000 UTC m=+593.178198619" Nov 03 10:38:36.107964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:36.107929 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:36.157779 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:36.157745 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:36.170456 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:36.170429 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:36.193006 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:36.192980 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5" Nov 03 10:38:40.208551 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:40.208517 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:40.221410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:40.221385 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:41.220615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:41.217354 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-bpt97" Nov 03 10:38:42.222576 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:42.222549 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:38:42.224881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:42.224857 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:38:42.227712 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:42.227691 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:38:42.229691 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:42.229671 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:38:42.267393 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:42.267367 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:42.279598 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:42.279574 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:43.227384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:43.227355 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4" Nov 03 10:38:44.107538 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:44.107496 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:44.121566 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:44.121539 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:44.236876 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:44.236847 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r" Nov 03 10:38:47.348243 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.348203 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b56ffbcf8-d4flt"] Nov 03 10:38:47.353096 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.353075 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" Nov 03 10:38:47.362920 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.362895 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b56ffbcf8-d4flt"] Nov 03 10:38:47.458489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.458447 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac3c3232-9196-41f4-88ab-ffa55948e449-tls-cert\") pod \"authorino-5b56ffbcf8-d4flt\" (UID: \"ac3c3232-9196-41f4-88ab-ffa55948e449\") " pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" Nov 03 10:38:47.458489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.458495 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgff\" (UniqueName: \"kubernetes.io/projected/ac3c3232-9196-41f4-88ab-ffa55948e449-kube-api-access-gkgff\") pod \"authorino-5b56ffbcf8-d4flt\" (UID: \"ac3c3232-9196-41f4-88ab-ffa55948e449\") " pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" Nov 03 10:38:47.560016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.559978 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac3c3232-9196-41f4-88ab-ffa55948e449-tls-cert\") pod \"authorino-5b56ffbcf8-d4flt\" (UID: \"ac3c3232-9196-41f4-88ab-ffa55948e449\") " pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" Nov 03 10:38:47.560016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.560016 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgff\" (UniqueName: \"kubernetes.io/projected/ac3c3232-9196-41f4-88ab-ffa55948e449-kube-api-access-gkgff\") pod \"authorino-5b56ffbcf8-d4flt\" (UID: \"ac3c3232-9196-41f4-88ab-ffa55948e449\") " pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" Nov 03 10:38:47.562479 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.562456 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac3c3232-9196-41f4-88ab-ffa55948e449-tls-cert\") pod \"authorino-5b56ffbcf8-d4flt\" (UID: \"ac3c3232-9196-41f4-88ab-ffa55948e449\") " pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" Nov 03 10:38:47.567657 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.567635 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgff\" (UniqueName: \"kubernetes.io/projected/ac3c3232-9196-41f4-88ab-ffa55948e449-kube-api-access-gkgff\") pod \"authorino-5b56ffbcf8-d4flt\" (UID: \"ac3c3232-9196-41f4-88ab-ffa55948e449\") " pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" Nov 03 10:38:47.665656 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.665571 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" Nov 03 10:38:47.936179 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:47.936151 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b56ffbcf8-d4flt"] Nov 03 10:38:47.938286 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:38:47.938257 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac3c3232_9196_41f4_88ab_ffa55948e449.slice/crio-47a9b1feb9faa3cfbf2aabf20347424a3fc844013ae20865b74461b59a736553 WatchSource:0}: Error finding container 47a9b1feb9faa3cfbf2aabf20347424a3fc844013ae20865b74461b59a736553: Status 404 returned error can't find the container with id 47a9b1feb9faa3cfbf2aabf20347424a3fc844013ae20865b74461b59a736553 Nov 03 10:38:48.244165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:48.244079 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" event={"ID":"ac3c3232-9196-41f4-88ab-ffa55948e449","Type":"ContainerStarted","Data":"47a9b1feb9faa3cfbf2aabf20347424a3fc844013ae20865b74461b59a736553"} Nov 03 10:38:49.266533 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.266492 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" event={"ID":"ac3c3232-9196-41f4-88ab-ffa55948e449","Type":"ContainerStarted","Data":"21e5ff0d5acb7c1a0af09189b4bf70e6222521b2491eedbbf777a64cc7c00dc7"} Nov 03 10:38:49.281031 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.280977 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b56ffbcf8-d4flt" podStartSLOduration=1.120006064 podStartE2EDuration="2.280963948s" totalImagesPullingTime="1.160957884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:49.279686453 +0000 UTC m=+607.264557998" watchObservedRunningTime="2026-11-03 10:38:49.280963948 +0000 UTC m=+607.265835230" Nov 03 10:38:49.305143 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.305107 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c989b787c-nvj7f"] Nov 03 10:38:49.305414 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.305386 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-c989b787c-nvj7f" podUID="00f207cd-eefa-430a-a7e9-36a30dc99f16" containerName="authorino" containerID="cri-o://8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b" gracePeriod=30 Nov 03 10:38:49.567236 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.567209 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:38:49.679676 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.679591 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f207cd-eefa-430a-a7e9-36a30dc99f16-kube-api-access-tmmrw\" (UniqueName: \"kubernetes.io/projected/00f207cd-eefa-430a-a7e9-36a30dc99f16-kube-api-access-tmmrw\") pod \"00f207cd-eefa-430a-a7e9-36a30dc99f16\" (UID: \"00f207cd-eefa-430a-a7e9-36a30dc99f16\") " Nov 03 10:38:49.679676 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.679643 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f207cd-eefa-430a-a7e9-36a30dc99f16-tls-cert\" (UniqueName: \"kubernetes.io/secret/00f207cd-eefa-430a-a7e9-36a30dc99f16-tls-cert\") pod \"00f207cd-eefa-430a-a7e9-36a30dc99f16\" (UID: \"00f207cd-eefa-430a-a7e9-36a30dc99f16\") " Nov 03 10:38:49.681846 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.681810 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f207cd-eefa-430a-a7e9-36a30dc99f16-kube-api-access-tmmrw" pod "00f207cd-eefa-430a-a7e9-36a30dc99f16" (UID: "00f207cd-eefa-430a-a7e9-36a30dc99f16"). InnerVolumeSpecName "kube-api-access-tmmrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:38:49.691468 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.691437 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f207cd-eefa-430a-a7e9-36a30dc99f16-tls-cert" pod "00f207cd-eefa-430a-a7e9-36a30dc99f16" (UID: "00f207cd-eefa-430a-a7e9-36a30dc99f16"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:38:49.781320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.781256 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmmrw\" (UniqueName: \"kubernetes.io/projected/00f207cd-eefa-430a-a7e9-36a30dc99f16-kube-api-access-tmmrw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:38:49.781493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:49.781333 5086 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f207cd-eefa-430a-a7e9-36a30dc99f16-tls-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:38:50.273052 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:50.273008 5086 generic.go:356] "Generic (PLEG): container finished" podID="00f207cd-eefa-430a-a7e9-36a30dc99f16" containerID="8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b" exitCode=0 Nov 03 10:38:50.273513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:50.273071 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c989b787c-nvj7f" Nov 03 10:38:50.273513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:50.273134 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c989b787c-nvj7f" event={"ID":"00f207cd-eefa-430a-a7e9-36a30dc99f16","Type":"ContainerDied","Data":"8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b"} Nov 03 10:38:50.273513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:50.273178 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c989b787c-nvj7f" event={"ID":"00f207cd-eefa-430a-a7e9-36a30dc99f16","Type":"ContainerDied","Data":"a01c285fc6eb0ff34a08d728ae9cec39e1c428fd150b2dd814918137bdfb7352"} Nov 03 10:38:50.273513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:50.273201 5086 scope.go:122] "RemoveContainer" containerID="8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b" Nov 03 10:38:50.283363 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:50.283340 5086 scope.go:122] "RemoveContainer" containerID="8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b" Nov 03 10:38:50.283686 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:38:50.283642 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b\": container with ID starting with 8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b not found: ID does not exist" containerID="8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b" Nov 03 10:38:50.283775 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:50.283682 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b"} err="failed to get container status \"8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b\": rpc error: code = NotFound desc = could not find container \"8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b\": container with ID starting with 8a4e4dc3da9f87a7e6cfdc1fc96a3a4a0bf34d042b8dd501392fe0e8591bef5b not found: ID does not exist" Nov 03 10:38:50.291976 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:50.291949 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c989b787c-nvj7f"] Nov 03 10:38:50.295634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:50.295608 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c989b787c-nvj7f"] Nov 03 10:38:52.169373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:52.169343 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f207cd-eefa-430a-a7e9-36a30dc99f16" path="/var/lib/kubelet/pods/00f207cd-eefa-430a-a7e9-36a30dc99f16/volumes" Nov 03 10:38:54.185913 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.185884 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77"] Nov 03 10:38:54.186371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.186312 5086 cpu_manager.go:414] "containerMap: removing container" podUID="00f207cd-eefa-430a-a7e9-36a30dc99f16" containerName="authorino" Nov 03 10:38:54.186371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.186329 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f207cd-eefa-430a-a7e9-36a30dc99f16" containerName="authorino" Nov 03 10:38:54.186451 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.186431 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f207cd-eefa-430a-a7e9-36a30dc99f16" containerName="authorino" Nov 03 10:38:54.191171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.191134 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.201742 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.200919 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 10:38:54.202012 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.201894 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rhbxr\"" Nov 03 10:38:54.205346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.205283 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77"] Nov 03 10:38:54.272870 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:38:54.272833 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:38:54.325671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.325636 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.325884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.325696 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.325884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.325784 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.325884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.325847 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.326018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.325923 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.326018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.326003 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.326099 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.326033 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cccpp\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-kube-api-access-cccpp\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.326161 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.326146 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.326196 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.326173 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.426937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.426896 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.426966 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.426995 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427020 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccpp\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-kube-api-access-cccpp\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427084 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427109 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427465 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427143 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427465 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427181 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427465 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427206 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427465 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427396 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427634 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427781 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.427813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.427802 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.428056 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.428029 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.430176 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.430125 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.439721 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.439662 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.442194 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.442167 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.442553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.442523 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccpp\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-kube-api-access-cccpp\") pod \"maas-default-gateway-openshift-default-6cc458d887-tfb77\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.529210 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.529179 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:54.814655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.814531 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77"] Nov 03 10:38:54.816347 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:38:54.816288 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7a978d_5cfe_4b97_94ea_6b825fa2eac4.slice/crio-8931d6eef0525a94fb5c3c9eb4327755d415b89c891fe7f889909fe49b2c839f WatchSource:0}: Error finding container 8931d6eef0525a94fb5c3c9eb4327755d415b89c891fe7f889909fe49b2c839f: Status 404 returned error can't find the container with id 8931d6eef0525a94fb5c3c9eb4327755d415b89c891fe7f889909fe49b2c839f Nov 03 10:38:54.819106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:54.819085 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:38:55.301507 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:55.301470 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" event={"ID":"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4","Type":"ContainerStarted","Data":"8931d6eef0525a94fb5c3c9eb4327755d415b89c891fe7f889909fe49b2c839f"} Nov 03 10:38:58.256025 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:58.255991 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:38:58.256362 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:58.256067 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:38:58.256362 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:58.256097 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:38:59.347792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:59.347758 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" event={"ID":"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4","Type":"ContainerStarted","Data":"0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80"} Nov 03 10:38:59.366597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:59.366545 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" podStartSLOduration=1.930956721 podStartE2EDuration="5.366522997s" totalImagesPullingTime="3.435566276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:59.365035948 +0000 UTC m=+617.349907232" watchObservedRunningTime="2026-11-03 10:38:59.366522997 +0000 UTC m=+617.351394281" Nov 03 10:38:59.540704 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:59.540664 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:38:59.541797 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:59.541769 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tfb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 10:38:59.541912 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:38:59.541825 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 10:39:00.531761 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:00.531719 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tfb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 10:39:00.532419 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:00.532387 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 10:39:01.536321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:01.534426 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tfb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 10:39:01.536321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:01.534498 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 10:39:02.537746 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:02.537704 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tfb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 10:39:02.538399 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:02.538360 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 10:39:03.540709 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:03.540671 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tfb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 10:39:03.541099 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:03.540740 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 10:39:04.530282 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:04.530245 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tfb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 10:39:04.530492 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:04.530349 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 10:39:05.530558 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:05.530520 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tfb77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 10:39:05.531048 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:05.530589 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 10:39:06.533996 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:06.533966 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:39:06.534385 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:06.534348 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:39:06.535024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:06.535006 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 10:39:09.200797 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.200761 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj"] Nov 03 10:39:09.206343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.206316 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.218968 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.218938 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj"] Nov 03 10:39:09.228000 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.227976 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr"] Nov 03 10:39:09.232415 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.232390 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.243454 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.243426 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr"] Nov 03 10:39:09.298519 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.298492 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7804ed8-ba49-4936-99f0-f02253369475-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.298519 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.298522 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7804ed8-ba49-4936-99f0-f02253369475-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.298725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.298544 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.298725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.298568 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7804ed8-ba49-4936-99f0-f02253369475-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.298725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.298701 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjkf\" (UniqueName: \"kubernetes.io/projected/b7804ed8-ba49-4936-99f0-f02253369475-kube-api-access-rzjkf\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.298823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.298738 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.298823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.298763 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.298823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.298777 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.298823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.298802 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.399768 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.399733 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjkf\" (UniqueName: \"kubernetes.io/projected/b7804ed8-ba49-4936-99f0-f02253369475-kube-api-access-rzjkf\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.399768 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.399772 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.400019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.399791 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.400019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.399812 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9fg\" (UniqueName: \"kubernetes.io/projected/99bbba2f-874c-4246-9ec3-e92e117a4159-kube-api-access-gk9fg\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.400019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.399979 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400003 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.400237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400038 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.400237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400082 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400117 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400208 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400234 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.400548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400279 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99bbba2f-874c-4246-9ec3-e92e117a4159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.400548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400376 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.400548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400457 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400461 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.400548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400515 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400544 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400584 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7804ed8-ba49-4936-99f0-f02253369475-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400620 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7804ed8-ba49-4936-99f0-f02253369475-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400654 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.400813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400677 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7804ed8-ba49-4936-99f0-f02253369475-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.401047 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.400952 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.401555 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.401528 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7804ed8-ba49-4936-99f0-f02253369475-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.402694 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.402673 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7804ed8-ba49-4936-99f0-f02253369475-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.402999 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.402971 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7804ed8-ba49-4936-99f0-f02253369475-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.407889 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.407863 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7804ed8-ba49-4936-99f0-f02253369475-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.407991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.407873 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjkf\" (UniqueName: \"kubernetes.io/projected/b7804ed8-ba49-4936-99f0-f02253369475-kube-api-access-rzjkf\") pod \"maas-default-gateway-openshift-default-6cc458d887-5p4bj\" (UID: \"b7804ed8-ba49-4936-99f0-f02253369475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.502339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.502217 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.502339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.502256 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.502339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.502284 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9fg\" (UniqueName: \"kubernetes.io/projected/99bbba2f-874c-4246-9ec3-e92e117a4159-kube-api-access-gk9fg\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.502795 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.502769 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.502926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.502904 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.503002 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.502945 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.503002 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.502984 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.503107 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.503022 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99bbba2f-874c-4246-9ec3-e92e117a4159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.503107 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.503064 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.503207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.503115 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.503207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.503138 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.503289 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.503233 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.503354 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.503343 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.503876 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.503848 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99bbba2f-874c-4246-9ec3-e92e117a4159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.505085 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.505060 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.505167 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.505067 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.510142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.510115 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9fg\" (UniqueName: \"kubernetes.io/projected/99bbba2f-874c-4246-9ec3-e92e117a4159-kube-api-access-gk9fg\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.510373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.510329 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99bbba2f-874c-4246-9ec3-e92e117a4159-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-lrmsr\" (UID: \"99bbba2f-874c-4246-9ec3-e92e117a4159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.535455 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.535429 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:09.551636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.551615 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:09.816711 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.816686 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj"] Nov 03 10:39:09.818274 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:39:09.818247 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7804ed8_ba49_4936_99f0_f02253369475.slice/crio-c24409d8e486be44bc10605bbba83efd2fd65e013431a429f35cb36df2048373 WatchSource:0}: Error finding container c24409d8e486be44bc10605bbba83efd2fd65e013431a429f35cb36df2048373: Status 404 returned error can't find the container with id c24409d8e486be44bc10605bbba83efd2fd65e013431a429f35cb36df2048373 Nov 03 10:39:09.821962 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.821928 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:39:09.822041 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.821996 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:39:09.822041 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.822024 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:39:09.838720 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.838697 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr"] Nov 03 10:39:09.841775 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:39:09.841748 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bbba2f_874c_4246_9ec3_e92e117a4159.slice/crio-c6c5684465a8a42e1062208cfeaa7d6fc86f62e6630ef11cb90e68947224e466 WatchSource:0}: Error finding container c6c5684465a8a42e1062208cfeaa7d6fc86f62e6630ef11cb90e68947224e466: Status 404 returned error can't find the container with id c6c5684465a8a42e1062208cfeaa7d6fc86f62e6630ef11cb90e68947224e466 Nov 03 10:39:09.844724 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.844696 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:39:09.844805 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.844755 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:39:09.844805 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:09.844785 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:39:10.402630 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.402588 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" event={"ID":"b7804ed8-ba49-4936-99f0-f02253369475","Type":"ContainerStarted","Data":"f25e65deeced72d753430a3645396c351ea924cec68b7e0086b868e05dd873cf"} Nov 03 10:39:10.403014 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.402652 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" event={"ID":"b7804ed8-ba49-4936-99f0-f02253369475","Type":"ContainerStarted","Data":"c24409d8e486be44bc10605bbba83efd2fd65e013431a429f35cb36df2048373"} Nov 03 10:39:10.404855 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.404820 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" event={"ID":"99bbba2f-874c-4246-9ec3-e92e117a4159","Type":"ContainerStarted","Data":"9e1b0c5dc74064c6a3070d9b35a7ba11205eed5c2336e3818fd4e659c7527d5f"} Nov 03 10:39:10.404993 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.404872 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" event={"ID":"99bbba2f-874c-4246-9ec3-e92e117a4159","Type":"ContainerStarted","Data":"c6c5684465a8a42e1062208cfeaa7d6fc86f62e6630ef11cb90e68947224e466"} Nov 03 10:39:10.435318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.434288 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podStartSLOduration=1.434266428 podStartE2EDuration="1.434266428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:39:10.425938234 +0000 UTC m=+628.410809519" watchObservedRunningTime="2026-11-03 10:39:10.434266428 +0000 UTC m=+628.419137714" Nov 03 10:39:10.542318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.540345 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:10.546318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.544412 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5p4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 10:39:10.546318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.544475 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podUID="b7804ed8-ba49-4936-99f0-f02253369475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 10:39:10.563318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.554411 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:10.563318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.554576 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lrmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 10:39:10.563318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:10.554621 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podUID="99bbba2f-874c-4246-9ec3-e92e117a4159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 10:39:11.537321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:11.536670 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5p4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 10:39:11.537321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:11.536736 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podUID="b7804ed8-ba49-4936-99f0-f02253369475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 10:39:11.554723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:11.554690 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lrmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 10:39:11.554982 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:11.554948 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podUID="99bbba2f-874c-4246-9ec3-e92e117a4159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 10:39:12.540326 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:12.539673 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5p4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 10:39:12.540326 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:12.539752 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podUID="b7804ed8-ba49-4936-99f0-f02253369475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 10:39:12.553354 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:12.553322 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lrmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 10:39:12.553638 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:12.553603 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podUID="99bbba2f-874c-4246-9ec3-e92e117a4159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 10:39:13.539322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:13.538727 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5p4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 10:39:13.539322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:13.538794 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podUID="b7804ed8-ba49-4936-99f0-f02253369475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 10:39:13.552663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:13.552627 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lrmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 10:39:13.553244 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:13.553216 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podUID="99bbba2f-874c-4246-9ec3-e92e117a4159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 10:39:14.559320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:14.542734 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5p4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 10:39:14.559320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:14.542816 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podUID="b7804ed8-ba49-4936-99f0-f02253369475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 10:39:14.559320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:14.554935 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lrmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 10:39:14.559320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:14.555003 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podUID="99bbba2f-874c-4246-9ec3-e92e117a4159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 10:39:15.540331 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:15.539742 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5p4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 10:39:15.540331 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:15.539805 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podUID="b7804ed8-ba49-4936-99f0-f02253369475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 10:39:15.559319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:15.555952 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lrmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 10:39:15.559319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:15.556012 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podUID="99bbba2f-874c-4246-9ec3-e92e117a4159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 10:39:16.536642 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:16.536603 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5p4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 10:39:16.536820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:16.536675 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podUID="b7804ed8-ba49-4936-99f0-f02253369475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 10:39:16.552250 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:16.552214 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lrmsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 10:39:16.552371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:16.552277 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podUID="99bbba2f-874c-4246-9ec3-e92e117a4159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 10:39:18.177617 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:18.177585 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:18.178352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:18.178329 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:18.179816 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:18.179794 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" Nov 03 10:39:18.192792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:18.192756 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:18.193083 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:18.193059 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:18.193829 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:18.193811 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" Nov 03 10:39:18.196712 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:39:18.196659 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podStartSLOduration=9.196645663 podStartE2EDuration="9.196645663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:39:10.456135445 +0000 UTC m=+628.441006729" watchObservedRunningTime="2026-11-03 10:39:18.196645663 +0000 UTC m=+636.181516949" Nov 03 10:39:24.274308 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:39:24.274257 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:39:54.273524 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:39:54.273431 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:40:01.652646 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.652606 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp"] Nov 03 10:40:01.656789 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.656761 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.672446 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.672415 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp"] Nov 03 10:40:01.733306 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.732710 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3baa762a-gw-openshift-default-dockercfg-8vf7x\"" Nov 03 10:40:01.759822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.759775 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-token\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.760101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.759870 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccfjx\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-kube-api-access-ccfjx\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.760101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.759919 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-certs\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.760101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.759963 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-podinfo\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.760101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.760006 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-credential-socket\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.760101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.760035 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-socket\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.760101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.760068 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-data\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.760101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.760104 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-envoy\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.760532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.760134 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb0ba09-173d-44e7-8215-ce4950c33a41-istiod-ca-cert\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.861236 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.861180 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-envoy\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.861236 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.861235 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb0ba09-173d-44e7-8215-ce4950c33a41-istiod-ca-cert\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.861521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.861349 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-token\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.861521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.861422 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccfjx\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-kube-api-access-ccfjx\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.861521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.861494 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-certs\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.861678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.861570 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-podinfo\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.861678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.861623 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-credential-socket\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.861678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.861652 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-socket\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.861836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.861693 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-data\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.862170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.862130 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-data\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.862170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.862156 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-certs\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.862331 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.862147 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb0ba09-173d-44e7-8215-ce4950c33a41-istiod-ca-cert\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.862519 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.862499 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-socket\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.862627 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.862517 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-credential-socket\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.864535 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.864515 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-envoy\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.868420 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.868396 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-podinfo\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.873504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.873481 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-token\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.874494 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.874470 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccfjx\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-kube-api-access-ccfjx\") pod \"e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:01.973349 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:01.973254 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:02.257928 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:02.257904 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp"] Nov 03 10:40:02.260071 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:02.260047 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb0ba09_173d_44e7_8215_ce4950c33a41.slice/crio-96f4fe0b5d9e0811482f4ba392a6529426e2164be5fda2f81ca0171c9d96d10c WatchSource:0}: Error finding container 96f4fe0b5d9e0811482f4ba392a6529426e2164be5fda2f81ca0171c9d96d10c: Status 404 returned error can't find the container with id 96f4fe0b5d9e0811482f4ba392a6529426e2164be5fda2f81ca0171c9d96d10c Nov 03 10:40:02.262386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:02.262353 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:02.262451 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:02.262427 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:02.262488 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:02.262458 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:02.724913 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:02.724873 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" event={"ID":"abb0ba09-173d-44e7-8215-ce4950c33a41","Type":"ContainerStarted","Data":"e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24"} Nov 03 10:40:02.725323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:02.724941 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" event={"ID":"abb0ba09-173d-44e7-8215-ce4950c33a41","Type":"ContainerStarted","Data":"96f4fe0b5d9e0811482f4ba392a6529426e2164be5fda2f81ca0171c9d96d10c"} Nov 03 10:40:02.747602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:02.747554 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" podStartSLOduration=1.747539874 podStartE2EDuration="1.747539874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:02.746786356 +0000 UTC m=+680.731657650" watchObservedRunningTime="2026-11-03 10:40:02.747539874 +0000 UTC m=+680.732411246" Nov 03 10:40:02.974376 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:02.974337 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:03.979718 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:03.979689 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:04.735754 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:04.735724 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:04.736778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:04.736753 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:05.567909 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.567868 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm"] Nov 03 10:40:05.571896 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.571878 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:05.580060 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.579453 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3a6822cf\"" Nov 03 10:40:05.582204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.582175 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm"] Nov 03 10:40:05.699542 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.699506 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe3241e-5fae-4156-aaa1-d514d65f9917-plugins-config-volume\") pod \"payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm\" (UID: \"fbe3241e-5fae-4156-aaa1-d514d65f9917\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:05.699723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.699663 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6f6c\" (UniqueName: \"kubernetes.io/projected/fbe3241e-5fae-4156-aaa1-d514d65f9917-kube-api-access-r6f6c\") pod \"payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm\" (UID: \"fbe3241e-5fae-4156-aaa1-d514d65f9917\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:05.800987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.800952 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe3241e-5fae-4156-aaa1-d514d65f9917-plugins-config-volume\") pod \"payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm\" (UID: \"fbe3241e-5fae-4156-aaa1-d514d65f9917\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:05.801181 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.801078 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6f6c\" (UniqueName: \"kubernetes.io/projected/fbe3241e-5fae-4156-aaa1-d514d65f9917-kube-api-access-r6f6c\") pod \"payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm\" (UID: \"fbe3241e-5fae-4156-aaa1-d514d65f9917\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:05.801786 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.801765 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe3241e-5fae-4156-aaa1-d514d65f9917-plugins-config-volume\") pod \"payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm\" (UID: \"fbe3241e-5fae-4156-aaa1-d514d65f9917\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:05.812080 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.812050 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6f6c\" (UniqueName: \"kubernetes.io/projected/fbe3241e-5fae-4156-aaa1-d514d65f9917-kube-api-access-r6f6c\") pod \"payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm\" (UID: \"fbe3241e-5fae-4156-aaa1-d514d65f9917\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:05.887852 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:05.887773 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:06.161043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.161017 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm"] Nov 03 10:40:06.163699 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:06.163656 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe3241e_5fae_4156_aaa1_d514d65f9917.slice/crio-12f500fc3e962ae28a4ba3950338aaeac3a4b8c3ff2364d28b4190092cf41e35 WatchSource:0}: Error finding container 12f500fc3e962ae28a4ba3950338aaeac3a4b8c3ff2364d28b4190092cf41e35: Status 404 returned error can't find the container with id 12f500fc3e962ae28a4ba3950338aaeac3a4b8c3ff2364d28b4190092cf41e35 Nov 03 10:40:06.747458 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.747423 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" event={"ID":"fbe3241e-5fae-4156-aaa1-d514d65f9917","Type":"ContainerStarted","Data":"12f500fc3e962ae28a4ba3950338aaeac3a4b8c3ff2364d28b4190092cf41e35"} Nov 03 10:40:06.949221 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.949182 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48"] Nov 03 10:40:06.954221 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.954195 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:06.963372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.963192 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 10:40:06.963839 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.963723 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 10:40:06.963839 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.963792 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3a6822cf-serving-cert\"" Nov 03 10:40:06.965546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.965276 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 10:40:06.966314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.966258 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l8bvs\"" Nov 03 10:40:06.978232 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:06.977733 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48"] Nov 03 10:40:07.015065 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.014978 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-metrics-certs\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.015354 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.015315 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-maas-api-tls\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.015511 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.015483 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5k2\" (UniqueName: \"kubernetes.io/projected/adb92568-0036-4ee3-b70a-6b096f117fa4-kube-api-access-vw5k2\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.015615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.015599 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb92568-0036-4ee3-b70a-6b096f117fa4-openshift-service-ca\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.117321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.116492 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-metrics-certs\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.117321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.116541 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-maas-api-tls\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.117321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.116585 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5k2\" (UniqueName: \"kubernetes.io/projected/adb92568-0036-4ee3-b70a-6b096f117fa4-kube-api-access-vw5k2\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.117321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.116685 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb92568-0036-4ee3-b70a-6b096f117fa4-openshift-service-ca\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.117738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.117486 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb92568-0036-4ee3-b70a-6b096f117fa4-openshift-service-ca\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.121712 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.121653 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-maas-api-tls\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.121872 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.121752 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-metrics-certs\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.128716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.128686 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5k2\" (UniqueName: \"kubernetes.io/projected/adb92568-0036-4ee3-b70a-6b096f117fa4-kube-api-access-vw5k2\") pod \"maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.283201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.283105 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:07.571488 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.571454 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48"] Nov 03 10:40:07.754940 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:07.754894 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" event={"ID":"adb92568-0036-4ee3-b70a-6b096f117fa4","Type":"ContainerStarted","Data":"991609075fba3d7caf69113dcba8b36b3bc525baa17aff2df6c93d38c0c91099"} Nov 03 10:40:09.871319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:09.870689 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" event={"ID":"fbe3241e-5fae-4156-aaa1-d514d65f9917","Type":"ContainerStarted","Data":"875f7245c4b356103595bc2527194a6705ae283a37dfb60eb5a941073b714669"} Nov 03 10:40:09.893320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:09.871887 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:10.880509 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:10.880473 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" event={"ID":"adb92568-0036-4ee3-b70a-6b096f117fa4","Type":"ContainerStarted","Data":"ec42b5e73573d3d49b16d97b3e68987755302c5919db13181cb046a0c4656207"} Nov 03 10:40:10.880960 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:10.880630 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:10.902797 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:10.902741 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" podStartSLOduration=2.7948018919999997 podStartE2EDuration="4.902725109s" totalImagesPullingTime="2.107923217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:10.900223661 +0000 UTC m=+688.885094944" watchObservedRunningTime="2026-11-03 10:40:10.902725109 +0000 UTC m=+688.887596400" Nov 03 10:40:10.902975 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:10.902952 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" podStartSLOduration=3.011566102 podStartE2EDuration="5.90294759s" totalImagesPullingTime="2.891381488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:09.897194548 +0000 UTC m=+687.882065831" watchObservedRunningTime="2026-11-03 10:40:10.90294759 +0000 UTC m=+688.887818872" Nov 03 10:40:11.356631 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.356592 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq"] Nov 03 10:40:11.360741 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.360721 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.367467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.367440 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-19708dae-openshift-default-dockercfg-kk4vd\"" Nov 03 10:40:11.375255 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.375230 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq"] Nov 03 10:40:11.489386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.489352 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj5mv\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-kube-api-access-xj5mv\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.489386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.489398 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istiod-ca-cert\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.489678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.489432 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-certs\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.489678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.489600 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-socket\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.489801 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.489657 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-token\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.489801 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.489725 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-credential-socket\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.489913 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.489853 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-envoy\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.489972 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.489940 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-data\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.490043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.489980 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-podinfo\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.590789 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.590730 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-envoy\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.590991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.590810 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-data\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.590991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.590844 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-podinfo\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.590991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.590889 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj5mv\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-kube-api-access-xj5mv\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.590991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.590921 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istiod-ca-cert\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.590991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.590952 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-certs\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.591254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.591006 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-socket\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.591254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.591032 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-token\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.591254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.591069 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-credential-socket\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.591254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.591244 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-data\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.591496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.591388 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-socket\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.591496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.591479 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-credential-socket\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.591579 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.591538 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-certs\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.591842 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.591825 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istiod-ca-cert\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.593236 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.593207 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-envoy\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.593461 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.593443 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-podinfo\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.601013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.600990 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-token\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.601132 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.601113 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj5mv\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-kube-api-access-xj5mv\") pod \"e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.677204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.677109 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:11.985982 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.985958 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq"] Nov 03 10:40:11.988456 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:11.988423 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f03a7ba_dcc4_4f79_b366_f7ce6f878f29.slice/crio-c1efdcd860c0149ebdfe9852cd3b911f6944b66251b3ed39fdfd14622f3f7f63 WatchSource:0}: Error finding container c1efdcd860c0149ebdfe9852cd3b911f6944b66251b3ed39fdfd14622f3f7f63: Status 404 returned error can't find the container with id c1efdcd860c0149ebdfe9852cd3b911f6944b66251b3ed39fdfd14622f3f7f63 Nov 03 10:40:11.991374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.991336 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:11.991489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.991427 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:11.991489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:11.991465 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:12.894310 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:12.894257 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" event={"ID":"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29","Type":"ContainerStarted","Data":"0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a"} Nov 03 10:40:12.894505 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:12.894328 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" event={"ID":"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29","Type":"ContainerStarted","Data":"c1efdcd860c0149ebdfe9852cd3b911f6944b66251b3ed39fdfd14622f3f7f63"} Nov 03 10:40:13.047328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.046686 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" podStartSLOduration=2.04666435 podStartE2EDuration="2.04666435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:12.944548983 +0000 UTC m=+690.929420293" watchObservedRunningTime="2026-11-03 10:40:13.04666435 +0000 UTC m=+691.031535634" Nov 03 10:40:13.052331 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.048282 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd"] Nov 03 10:40:13.055416 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.055391 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.067689 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.067639 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-afe85416-openshift-default-dockercfg-x4hh4\"" Nov 03 10:40:13.073204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.073176 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd"] Nov 03 10:40:13.109013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.108369 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istiod-ca-cert\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.109013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.108483 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-credential-socket\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.109013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.108530 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-envoy\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.109013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.108560 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-certs\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.109013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.108611 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-token\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.109013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.108647 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-data\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.109013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.108675 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-podinfo\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.109013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.108708 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-socket\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.109013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.108765 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdx7r\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-kube-api-access-kdx7r\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210051 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.209971 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-credential-socket\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210051 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210028 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-envoy\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210051 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210049 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-certs\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210168 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-token\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210207 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-data\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210235 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-podinfo\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210265 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-socket\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210327 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdx7r\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-kube-api-access-kdx7r\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210639 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210450 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istiod-ca-cert\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210639 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210584 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-credential-socket\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210788 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-certs\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.210881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.210860 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-data\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.211257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.211225 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-socket\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.211257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.211238 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istiod-ca-cert\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.213068 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.213038 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-envoy\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.213215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.213193 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-podinfo\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.239319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.234866 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-token\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.245650 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.242923 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdx7r\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-kube-api-access-kdx7r\") pod \"e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.372904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.372864 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:13.677613 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.677590 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:13.680328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.680274 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd"] Nov 03 10:40:13.680774 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:13.680747 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdda61d4_b6ba_47f3_a3a9_9fe9ead6bfa1.slice/crio-c9159a6cfa1eb40c1f04bfd2e8af0046d9f26929a9d66aa2fe62ab00d2d43a22 WatchSource:0}: Error finding container c9159a6cfa1eb40c1f04bfd2e8af0046d9f26929a9d66aa2fe62ab00d2d43a22: Status 404 returned error can't find the container with id c9159a6cfa1eb40c1f04bfd2e8af0046d9f26929a9d66aa2fe62ab00d2d43a22 Nov 03 10:40:13.683371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.683330 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:13.683482 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.683414 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:13.683482 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.683458 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:13.684631 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.684604 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:13.903223 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.903169 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" event={"ID":"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1","Type":"ContainerStarted","Data":"c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72"} Nov 03 10:40:13.903443 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.903267 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" event={"ID":"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1","Type":"ContainerStarted","Data":"c9159a6cfa1eb40c1f04bfd2e8af0046d9f26929a9d66aa2fe62ab00d2d43a22"} Nov 03 10:40:13.903684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.903656 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:13.904593 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.904573 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:13.923139 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:13.923084 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" podStartSLOduration=0.923070728 podStartE2EDuration="923.070728ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:13.922003909 +0000 UTC m=+691.906875195" watchObservedRunningTime="2026-11-03 10:40:13.923070728 +0000 UTC m=+691.907942010" Nov 03 10:40:14.373358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:14.373317 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:14.378595 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:14.378573 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:14.909618 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:14.909587 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:14.910712 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:14.910671 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:16.818358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:16.818323 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64"] Nov 03 10:40:16.822788 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:16.822762 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:16.837385 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:16.837354 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64"] Nov 03 10:40:16.882440 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:16.882404 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-afe85416\"" Nov 03 10:40:16.882807 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:16.882777 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-afe85416-dockercfg-zwxt9\"" Nov 03 10:40:16.894163 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:16.894138 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:16.966991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:16.966950 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69879\" (UniqueName: \"kubernetes.io/projected/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-kube-api-access-69879\") pod \"payload-processing-e2e-ait-afe85416-77455b99f-ffk64\" (UID: \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\") " pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:16.982120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:16.967125 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-plugins-config-volume\") pod \"payload-processing-e2e-ait-afe85416-77455b99f-ffk64\" (UID: \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\") " pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:17.068785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.068709 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-plugins-config-volume\") pod \"payload-processing-e2e-ait-afe85416-77455b99f-ffk64\" (UID: \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\") " pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:17.068922 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.068905 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69879\" (UniqueName: \"kubernetes.io/projected/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-kube-api-access-69879\") pod \"payload-processing-e2e-ait-afe85416-77455b99f-ffk64\" (UID: \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\") " pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:17.069792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.069769 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-plugins-config-volume\") pod \"payload-processing-e2e-ait-afe85416-77455b99f-ffk64\" (UID: \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\") " pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:17.081788 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.081759 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69879\" (UniqueName: \"kubernetes.io/projected/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-kube-api-access-69879\") pod \"payload-processing-e2e-ait-afe85416-77455b99f-ffk64\" (UID: \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\") " pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:17.147001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.146964 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:17.422172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.422074 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64"] Nov 03 10:40:17.424695 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:17.424668 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9ccf93_9d99_4aa1_a4fe_642731db5b00.slice/crio-ce6e9f8c62e7665ffcf91dd59070ecf2ee92a277c40d58ef0a53a4fe7495e0a8 WatchSource:0}: Error finding container ce6e9f8c62e7665ffcf91dd59070ecf2ee92a277c40d58ef0a53a4fe7495e0a8: Status 404 returned error can't find the container with id ce6e9f8c62e7665ffcf91dd59070ecf2ee92a277c40d58ef0a53a4fe7495e0a8 Nov 03 10:40:17.751698 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.751605 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4"] Nov 03 10:40:17.755755 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.755731 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.772635 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.772606 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4"] Nov 03 10:40:17.848705 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.848670 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-19708dae-serving-cert\"" Nov 03 10:40:17.878675 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.878637 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-metrics-certs\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.878884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.878685 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5wk5\" (UniqueName: \"kubernetes.io/projected/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-kube-api-access-w5wk5\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.878964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.878892 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-openshift-service-ca\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.879027 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.879009 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-maas-api-tls\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.937902 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.937870 5086 generic.go:356] "Generic (PLEG): container finished" podID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerID="e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24" exitCode=0 Nov 03 10:40:17.938088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.937946 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" event={"ID":"abb0ba09-173d-44e7-8215-ce4950c33a41","Type":"ContainerDied","Data":"e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24"} Nov 03 10:40:17.938430 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.938409 5086 scope.go:122] "RemoveContainer" containerID="e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24" Nov 03 10:40:17.939813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.939791 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" event={"ID":"ae9ccf93-9d99-4aa1-a4fe-642731db5b00","Type":"ContainerStarted","Data":"dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83"} Nov 03 10:40:17.939915 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.939832 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" event={"ID":"ae9ccf93-9d99-4aa1-a4fe-642731db5b00","Type":"ContainerStarted","Data":"ce6e9f8c62e7665ffcf91dd59070ecf2ee92a277c40d58ef0a53a4fe7495e0a8"} Nov 03 10:40:17.939915 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.939824 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:17.939915 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.939855 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:17.940100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.939912 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:17.940100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.939964 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:17.980013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.979953 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" podStartSLOduration=1.9799384199999999 podStartE2EDuration="1.97993842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:17.978122718 +0000 UTC m=+695.962994004" watchObservedRunningTime="2026-11-03 10:40:17.97993842 +0000 UTC m=+695.964809703" Nov 03 10:40:17.981507 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.981474 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-openshift-service-ca\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.981662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.981541 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-maas-api-tls\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.981662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.981605 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-metrics-certs\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.981662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.981642 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5wk5\" (UniqueName: \"kubernetes.io/projected/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-kube-api-access-w5wk5\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.982369 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.982339 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-openshift-service-ca\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.984375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.984343 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-maas-api-tls\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.984763 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.984745 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-metrics-certs\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:17.991138 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:17.991082 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5wk5\" (UniqueName: \"kubernetes.io/projected/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-kube-api-access-w5wk5\") pod \"maas-api-e2e-ait-19708dae-697d49db-ckmj4\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:18.076895 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:18.076864 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:18.365782 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:18.365752 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4"] Nov 03 10:40:18.368031 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:18.367988 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f26fb3_6e2f_490e_bc03_dd84358e3c34.slice/crio-3206e263940d21af57882717788d9cc80bd54f2ca8a475002a2e2d7631f6dde6 WatchSource:0}: Error finding container 3206e263940d21af57882717788d9cc80bd54f2ca8a475002a2e2d7631f6dde6: Status 404 returned error can't find the container with id 3206e263940d21af57882717788d9cc80bd54f2ca8a475002a2e2d7631f6dde6 Nov 03 10:40:18.950755 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:18.950721 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" event={"ID":"a5f26fb3-6e2f-490e-bc03-dd84358e3c34","Type":"ContainerStarted","Data":"3206e263940d21af57882717788d9cc80bd54f2ca8a475002a2e2d7631f6dde6"} Nov 03 10:40:18.952850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:18.952821 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" event={"ID":"abb0ba09-173d-44e7-8215-ce4950c33a41","Type":"ContainerStarted","Data":"330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc"} Nov 03 10:40:18.974117 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:18.974079 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:19.149787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.149692 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k"] Nov 03 10:40:19.154698 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.154676 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.167697 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.167653 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-afe85416-serving-cert\"" Nov 03 10:40:19.171290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.171265 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k"] Nov 03 10:40:19.297493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.297454 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bqt\" (UniqueName: \"kubernetes.io/projected/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-kube-api-access-j6bqt\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.297692 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.297514 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-metrics-certs\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.297692 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.297602 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-openshift-service-ca\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.297821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.297705 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-maas-api-tls\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.398361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.398291 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-maas-api-tls\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.398532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.398382 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bqt\" (UniqueName: \"kubernetes.io/projected/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-kube-api-access-j6bqt\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.398532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.398428 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-metrics-certs\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.398532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.398489 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-openshift-service-ca\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.399137 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.399106 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-openshift-service-ca\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.400736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.400684 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-maas-api-tls\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.400736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.400727 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-metrics-certs\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.405634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.405608 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bqt\" (UniqueName: \"kubernetes.io/projected/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-kube-api-access-j6bqt\") pod \"maas-api-e2e-ait-afe85416-7c48854c69-rxx2k\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.475651 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.475603 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:19.740373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.739406 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:19.775470 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.775444 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k"] Nov 03 10:40:19.777366 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:19.777339 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09396946_8b1b_44e9_a0e6_b78ea69e6d6a.slice/crio-53941a9247a256ef03a18d0bf78e52d9510c5edc01e8113e369be06e09904575 WatchSource:0}: Error finding container 53941a9247a256ef03a18d0bf78e52d9510c5edc01e8113e369be06e09904575: Status 404 returned error can't find the container with id 53941a9247a256ef03a18d0bf78e52d9510c5edc01e8113e369be06e09904575 Nov 03 10:40:19.960688 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.960648 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" event={"ID":"a5f26fb3-6e2f-490e-bc03-dd84358e3c34","Type":"ContainerStarted","Data":"936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63"} Nov 03 10:40:19.961155 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.960730 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:19.962001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.961975 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" event={"ID":"09396946-8b1b-44e9-a0e6-b78ea69e6d6a","Type":"ContainerStarted","Data":"53941a9247a256ef03a18d0bf78e52d9510c5edc01e8113e369be06e09904575"} Nov 03 10:40:19.974520 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.974494 5086 patch_prober.go:29] interesting pod/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 10:40:19.974618 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.974536 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 10:40:19.998621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:19.998525 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" podStartSLOduration=2.496092466 podStartE2EDuration="2.998510499s" totalImagesPullingTime="502.418033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:19.995792122 +0000 UTC m=+697.980663408" watchObservedRunningTime="2026-11-03 10:40:19.998510499 +0000 UTC m=+697.983381782" Nov 03 10:40:20.969001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:20.968959 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" event={"ID":"09396946-8b1b-44e9-a0e6-b78ea69e6d6a","Type":"ContainerStarted","Data":"1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783"} Nov 03 10:40:20.969648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:20.969197 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:20.974433 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:20.974409 5086 patch_prober.go:29] interesting pod/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 10:40:20.974518 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:20.974458 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 10:40:20.987446 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:20.987405 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" podStartSLOduration=1.499157205 podStartE2EDuration="1.987392542s" totalImagesPullingTime="488.235337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:20.984087895 +0000 UTC m=+698.968959175" watchObservedRunningTime="2026-11-03 10:40:20.987392542 +0000 UTC m=+698.972263836" Nov 03 10:40:21.888415 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:21.888385 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:21.973824 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:21.973793 5086 patch_prober.go:29] interesting pod/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 10:40:21.974310 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:21.973851 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 10:40:22.980468 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:22.980429 5086 patch_prober.go:29] interesting pod/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 10:40:22.981107 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:22.981072 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 10:40:23.976726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:23.976687 5086 patch_prober.go:29] interesting pod/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 10:40:23.977032 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:23.977005 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 10:40:24.298461 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:24.298354 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:40:24.980852 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:24.980819 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:24.985503 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:24.985476 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:25.976025 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:25.975984 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:26.981623 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:26.981594 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:28.957059 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:28.957028 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:29.946617 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:29.946524 5086 patch_prober.go:29] interesting pod/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:40:29.946617 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:29.946588 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:40:30.416564 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:30.405991 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp"] Nov 03 10:40:30.416564 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:30.406257 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" containerID="cri-o://330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc" gracePeriod=30 Nov 03 10:40:30.950496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:30.950452 5086 patch_prober.go:29] interesting pod/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:40:30.950657 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:30.950511 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:40:32.109627 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:32.109593 5086 generic.go:356] "Generic (PLEG): container finished" podID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerID="0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a" exitCode=0 Nov 03 10:40:32.110041 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:32.109650 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" event={"ID":"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29","Type":"ContainerDied","Data":"0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a"} Nov 03 10:40:32.110084 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:32.110039 5086 scope.go:122] "RemoveContainer" containerID="0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a" Nov 03 10:40:32.111460 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:32.111427 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:32.111550 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:32.111506 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:32.111589 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:32.111555 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:33.140323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.136997 5086 generic.go:356] "Generic (PLEG): container finished" podID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerID="c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72" exitCode=0 Nov 03 10:40:33.140323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.137141 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" event={"ID":"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1","Type":"ContainerDied","Data":"c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72"} Nov 03 10:40:33.140323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.137612 5086 scope.go:122] "RemoveContainer" containerID="c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72" Nov 03 10:40:33.140323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.139720 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:33.140323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.139810 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:33.140323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.139870 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:40:33.144191 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.144167 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" event={"ID":"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29","Type":"ContainerStarted","Data":"5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7"} Nov 03 10:40:33.677395 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.677356 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:33.677641 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.677597 5086 patch_prober.go:29] interesting pod/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 10:40:33.677717 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:33.677648 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 10:40:34.175921 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:34.175882 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" event={"ID":"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1","Type":"ContainerStarted","Data":"e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598"} Nov 03 10:40:34.374161 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:34.374020 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:34.374344 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:34.374307 5086 patch_prober.go:29] interesting pod/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 10:40:34.374393 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:34.374358 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 10:40:34.677809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:34.677772 5086 patch_prober.go:29] interesting pod/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 10:40:34.677990 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:34.677828 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 10:40:35.109142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.109107 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm"] Nov 03 10:40:35.373368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.373262 5086 patch_prober.go:29] interesting pod/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 10:40:35.373368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.373356 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 10:40:35.678919 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.678709 5086 patch_prober.go:29] interesting pod/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 10:40:35.678919 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.678767 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 10:40:35.912690 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.912656 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:35.996133 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.996047 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-token\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-token\") pod \"abb0ba09-173d-44e7-8215-ce4950c33a41\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " Nov 03 10:40:35.996133 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.996093 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-certs\") pod \"abb0ba09-173d-44e7-8215-ce4950c33a41\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " Nov 03 10:40:35.996133 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.996123 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-envoy\") pod \"abb0ba09-173d-44e7-8215-ce4950c33a41\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " Nov 03 10:40:35.996602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.996576 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-certs" pod "abb0ba09-173d-44e7-8215-ce4950c33a41" (UID: "abb0ba09-173d-44e7-8215-ce4950c33a41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:35.996798 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.996595 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-credential-socket\") pod \"abb0ba09-173d-44e7-8215-ce4950c33a41\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " Nov 03 10:40:35.996920 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.996907 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-socket\") pod \"abb0ba09-173d-44e7-8215-ce4950c33a41\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " Nov 03 10:40:35.997037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997025 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-kube-api-access-ccfjx\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-kube-api-access-ccfjx\") pod \"abb0ba09-173d-44e7-8215-ce4950c33a41\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " Nov 03 10:40:35.997188 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997171 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-data\") pod \"abb0ba09-173d-44e7-8215-ce4950c33a41\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " Nov 03 10:40:35.997410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.996816 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-credential-socket" pod "abb0ba09-173d-44e7-8215-ce4950c33a41" (UID: "abb0ba09-173d-44e7-8215-ce4950c33a41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:35.997410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997112 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-socket" pod "abb0ba09-173d-44e7-8215-ce4950c33a41" (UID: "abb0ba09-173d-44e7-8215-ce4950c33a41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:35.997410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997379 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-podinfo\") pod \"abb0ba09-173d-44e7-8215-ce4950c33a41\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " Nov 03 10:40:35.997410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997377 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-data" pod "abb0ba09-173d-44e7-8215-ce4950c33a41" (UID: "abb0ba09-173d-44e7-8215-ce4950c33a41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:35.997410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997409 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb0ba09-173d-44e7-8215-ce4950c33a41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb0ba09-173d-44e7-8215-ce4950c33a41-istiod-ca-cert\") pod \"abb0ba09-173d-44e7-8215-ce4950c33a41\" (UID: \"abb0ba09-173d-44e7-8215-ce4950c33a41\") " Nov 03 10:40:35.997837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997723 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:35.997837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997737 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:35.997837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997746 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:35.997837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997756 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:35.997837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.997802 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb0ba09-173d-44e7-8215-ce4950c33a41-istiod-ca-cert" pod "abb0ba09-173d-44e7-8215-ce4950c33a41" (UID: "abb0ba09-173d-44e7-8215-ce4950c33a41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:40:35.999158 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.999130 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-envoy" pod "abb0ba09-173d-44e7-8215-ce4950c33a41" (UID: "abb0ba09-173d-44e7-8215-ce4950c33a41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:35.999576 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.999557 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-token" pod "abb0ba09-173d-44e7-8215-ce4950c33a41" (UID: "abb0ba09-173d-44e7-8215-ce4950c33a41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:35.999681 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.999659 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-podinfo" pod "abb0ba09-173d-44e7-8215-ce4950c33a41" (UID: "abb0ba09-173d-44e7-8215-ce4950c33a41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:40:35.999733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:35.999660 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-kube-api-access-ccfjx" pod "abb0ba09-173d-44e7-8215-ce4950c33a41" (UID: "abb0ba09-173d-44e7-8215-ce4950c33a41"). InnerVolumeSpecName "kube-api-access-ccfjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:36.098775 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.098737 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccfjx\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-kube-api-access-ccfjx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:36.098775 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.098772 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:36.098992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.098785 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb0ba09-173d-44e7-8215-ce4950c33a41-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:36.098992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.098798 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:36.098992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.098811 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb0ba09-173d-44e7-8215-ce4950c33a41-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:36.292160 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.292076 5086 generic.go:356] "Generic (PLEG): container finished" podID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerID="330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc" exitCode=0 Nov 03 10:40:36.292337 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.292163 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" event={"ID":"abb0ba09-173d-44e7-8215-ce4950c33a41","Type":"ContainerDied","Data":"330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc"} Nov 03 10:40:36.292337 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.292226 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" event={"ID":"abb0ba09-173d-44e7-8215-ce4950c33a41","Type":"ContainerDied","Data":"96f4fe0b5d9e0811482f4ba392a6529426e2164be5fda2f81ca0171c9d96d10c"} Nov 03 10:40:36.292337 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.292247 5086 scope.go:122] "RemoveContainer" containerID="330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc" Nov 03 10:40:36.292457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.292403 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp" Nov 03 10:40:36.301822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.301791 5086 scope.go:122] "RemoveContainer" containerID="e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24" Nov 03 10:40:36.310521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.310503 5086 scope.go:122] "RemoveContainer" containerID="330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc" Nov 03 10:40:36.310806 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:36.310787 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc\": container with ID starting with 330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc not found: ID does not exist" containerID="330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc" Nov 03 10:40:36.310890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.310814 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc"} err="failed to get container status \"330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc\": rpc error: code = NotFound desc = could not find container \"330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc\": container with ID starting with 330dad1469d844a12fad6bf3987fe58086548b88cd23fd02de76a7c9439c8edc not found: ID does not exist" Nov 03 10:40:36.310890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.310832 5086 scope.go:122] "RemoveContainer" containerID="e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24" Nov 03 10:40:36.311106 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:36.311082 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24\": container with ID starting with e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24 not found: ID does not exist" containerID="e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24" Nov 03 10:40:36.311171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.311116 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24"} err="failed to get container status \"e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24\": rpc error: code = NotFound desc = could not find container \"e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24\": container with ID starting with e59bacf10b090139cdf18409d34f70ffcc3732d2c822c2f804cca20229cf0a24 not found: ID does not exist" Nov 03 10:40:36.343335 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.318098 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp"] Nov 03 10:40:36.343335 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.336095 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3baa762a-gw-openshift-default-6474774769-mkspp"] Nov 03 10:40:36.373715 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.373679 5086 patch_prober.go:29] interesting pod/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 10:40:36.374207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.373755 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 10:40:36.677885 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.677847 5086 patch_prober.go:29] interesting pod/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 10:40:36.678079 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:36.677914 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 10:40:37.382321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:37.381900 5086 patch_prober.go:29] interesting pod/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 10:40:37.382321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:37.381993 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 10:40:37.678458 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:37.678360 5086 patch_prober.go:29] interesting pod/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 10:40:37.678458 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:37.678442 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 10:40:38.171111 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:38.171073 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" path="/var/lib/kubelet/pods/abb0ba09-173d-44e7-8215-ce4950c33a41/volumes" Nov 03 10:40:38.373579 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:38.373545 5086 patch_prober.go:29] interesting pod/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 10:40:38.373762 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:38.373601 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 10:40:38.681668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:38.681630 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:38.682170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:38.682044 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:38.682795 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:38.682778 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:38.979428 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:38.979353 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48"] Nov 03 10:40:38.979791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:38.979761 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" podUID="adb92568-0036-4ee3-b70a-6b096f117fa4" containerName="maas-api" containerID="cri-o://ec42b5e73573d3d49b16d97b3e68987755302c5919db13181cb046a0c4656207" gracePeriod=30 Nov 03 10:40:39.321101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.321069 5086 generic.go:356] "Generic (PLEG): container finished" podID="adb92568-0036-4ee3-b70a-6b096f117fa4" containerID="ec42b5e73573d3d49b16d97b3e68987755302c5919db13181cb046a0c4656207" exitCode=0 Nov 03 10:40:39.321247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.321155 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" event={"ID":"adb92568-0036-4ee3-b70a-6b096f117fa4","Type":"ContainerDied","Data":"ec42b5e73573d3d49b16d97b3e68987755302c5919db13181cb046a0c4656207"} Nov 03 10:40:39.321247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.321220 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" event={"ID":"adb92568-0036-4ee3-b70a-6b096f117fa4","Type":"ContainerDied","Data":"991609075fba3d7caf69113dcba8b36b3bc525baa17aff2df6c93d38c0c91099"} Nov 03 10:40:39.321360 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.321249 5086 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="991609075fba3d7caf69113dcba8b36b3bc525baa17aff2df6c93d38c0c91099" Nov 03 10:40:39.329233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.329211 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:39.358655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.358613 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-metrics-certs\") pod \"adb92568-0036-4ee3-b70a-6b096f117fa4\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " Nov 03 10:40:39.358984 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.358965 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-maas-api-tls\") pod \"adb92568-0036-4ee3-b70a-6b096f117fa4\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " Nov 03 10:40:39.359151 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.359137 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb92568-0036-4ee3-b70a-6b096f117fa4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb92568-0036-4ee3-b70a-6b096f117fa4-openshift-service-ca\") pod \"adb92568-0036-4ee3-b70a-6b096f117fa4\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " Nov 03 10:40:39.359341 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.359327 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb92568-0036-4ee3-b70a-6b096f117fa4-kube-api-access-vw5k2\" (UniqueName: \"kubernetes.io/projected/adb92568-0036-4ee3-b70a-6b096f117fa4-kube-api-access-vw5k2\") pod \"adb92568-0036-4ee3-b70a-6b096f117fa4\" (UID: \"adb92568-0036-4ee3-b70a-6b096f117fa4\") " Nov 03 10:40:39.361753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.361721 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb92568-0036-4ee3-b70a-6b096f117fa4-openshift-service-ca" pod "adb92568-0036-4ee3-b70a-6b096f117fa4" (UID: "adb92568-0036-4ee3-b70a-6b096f117fa4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:40:39.363613 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.362586 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb92568-0036-4ee3-b70a-6b096f117fa4-kube-api-access-vw5k2" pod "adb92568-0036-4ee3-b70a-6b096f117fa4" (UID: "adb92568-0036-4ee3-b70a-6b096f117fa4"). InnerVolumeSpecName "kube-api-access-vw5k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:39.386358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.364583 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-maas-api-tls" pod "adb92568-0036-4ee3-b70a-6b096f117fa4" (UID: "adb92568-0036-4ee3-b70a-6b096f117fa4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:40:39.386358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.372457 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-metrics-certs" pod "adb92568-0036-4ee3-b70a-6b096f117fa4" (UID: "adb92568-0036-4ee3-b70a-6b096f117fa4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:40:39.451255 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.450937 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:39.451584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.451475 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:39.452751 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.452721 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:39.461428 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.461402 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb92568-0036-4ee3-b70a-6b096f117fa4-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:39.461553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.461430 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5k2\" (UniqueName: \"kubernetes.io/projected/adb92568-0036-4ee3-b70a-6b096f117fa4-kube-api-access-vw5k2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:39.461553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.461450 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:39.461553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:39.461462 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb92568-0036-4ee3-b70a-6b096f117fa4-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:40.110417 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.110377 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" podUID="fbe3241e-5fae-4156-aaa1-d514d65f9917" containerName="payload-processing" containerID="cri-o://875f7245c4b356103595bc2527194a6705ae283a37dfb60eb5a941073b714669" gracePeriod=25 Nov 03 10:40:40.341824 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.341791 5086 generic.go:356] "Generic (PLEG): container finished" podID="fbe3241e-5fae-4156-aaa1-d514d65f9917" containerID="875f7245c4b356103595bc2527194a6705ae283a37dfb60eb5a941073b714669" exitCode=0 Nov 03 10:40:40.341993 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.341884 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" event={"ID":"fbe3241e-5fae-4156-aaa1-d514d65f9917","Type":"ContainerDied","Data":"875f7245c4b356103595bc2527194a6705ae283a37dfb60eb5a941073b714669"} Nov 03 10:40:40.342252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.342234 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48" Nov 03 10:40:40.365346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.364224 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48"] Nov 03 10:40:40.367938 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.367914 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6822cf-5b6c5b974-jrf48"] Nov 03 10:40:40.387390 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.387366 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:40.472514 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.472471 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe3241e-5fae-4156-aaa1-d514d65f9917-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe3241e-5fae-4156-aaa1-d514d65f9917-plugins-config-volume\") pod \"fbe3241e-5fae-4156-aaa1-d514d65f9917\" (UID: \"fbe3241e-5fae-4156-aaa1-d514d65f9917\") " Nov 03 10:40:40.472688 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.472666 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe3241e-5fae-4156-aaa1-d514d65f9917-kube-api-access-r6f6c\" (UniqueName: \"kubernetes.io/projected/fbe3241e-5fae-4156-aaa1-d514d65f9917-kube-api-access-r6f6c\") pod \"fbe3241e-5fae-4156-aaa1-d514d65f9917\" (UID: \"fbe3241e-5fae-4156-aaa1-d514d65f9917\") " Nov 03 10:40:40.472896 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.472872 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe3241e-5fae-4156-aaa1-d514d65f9917-plugins-config-volume" pod "fbe3241e-5fae-4156-aaa1-d514d65f9917" (UID: "fbe3241e-5fae-4156-aaa1-d514d65f9917"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:40:40.474971 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.474944 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe3241e-5fae-4156-aaa1-d514d65f9917-kube-api-access-r6f6c" pod "fbe3241e-5fae-4156-aaa1-d514d65f9917" (UID: "fbe3241e-5fae-4156-aaa1-d514d65f9917"). InnerVolumeSpecName "kube-api-access-r6f6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:40.574209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.574129 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6f6c\" (UniqueName: \"kubernetes.io/projected/fbe3241e-5fae-4156-aaa1-d514d65f9917-kube-api-access-r6f6c\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:40.574521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:40.574387 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe3241e-5fae-4156-aaa1-d514d65f9917-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:41.349112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:41.349081 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" Nov 03 10:40:41.349605 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:41.349113 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm" event={"ID":"fbe3241e-5fae-4156-aaa1-d514d65f9917","Type":"ContainerDied","Data":"12f500fc3e962ae28a4ba3950338aaeac3a4b8c3ff2364d28b4190092cf41e35"} Nov 03 10:40:41.349605 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:41.349184 5086 scope.go:122] "RemoveContainer" containerID="875f7245c4b356103595bc2527194a6705ae283a37dfb60eb5a941073b714669" Nov 03 10:40:41.373062 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:41.373025 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm"] Nov 03 10:40:41.377445 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:41.377414 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6822cf-7b547f6cd9-jfvvm"] Nov 03 10:40:41.428830 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:41.428799 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64"] Nov 03 10:40:41.963277 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:41.963219 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k"] Nov 03 10:40:41.963779 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:41.963726 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" podUID="09396946-8b1b-44e9-a0e6-b78ea69e6d6a" containerName="maas-api" containerID="cri-o://1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783" gracePeriod=30 Nov 03 10:40:41.977668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:41.977636 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" podUID="09396946-8b1b-44e9-a0e6-b78ea69e6d6a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.66:8443/health\": dial tcp 10.132.0.66:8443: connect: connection refused" Nov 03 10:40:42.179020 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.178988 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb92568-0036-4ee3-b70a-6b096f117fa4" path="/var/lib/kubelet/pods/adb92568-0036-4ee3-b70a-6b096f117fa4/volumes" Nov 03 10:40:42.179785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.179758 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe3241e-5fae-4156-aaa1-d514d65f9917" path="/var/lib/kubelet/pods/fbe3241e-5fae-4156-aaa1-d514d65f9917/volumes" Nov 03 10:40:42.257663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.257633 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:42.296203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.294531 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-openshift-service-ca\") pod \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " Nov 03 10:40:42.296203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.294596 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-metrics-certs\") pod \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " Nov 03 10:40:42.296203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.294665 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-maas-api-tls\") pod \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " Nov 03 10:40:42.296203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.294848 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-kube-api-access-j6bqt\" (UniqueName: \"kubernetes.io/projected/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-kube-api-access-j6bqt\") pod \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\" (UID: \"09396946-8b1b-44e9-a0e6-b78ea69e6d6a\") " Nov 03 10:40:42.296203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.295535 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-openshift-service-ca" pod "09396946-8b1b-44e9-a0e6-b78ea69e6d6a" (UID: "09396946-8b1b-44e9-a0e6-b78ea69e6d6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:40:42.302729 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.302458 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-kube-api-access-j6bqt" pod "09396946-8b1b-44e9-a0e6-b78ea69e6d6a" (UID: "09396946-8b1b-44e9-a0e6-b78ea69e6d6a"). InnerVolumeSpecName "kube-api-access-j6bqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:42.302877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.302824 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-maas-api-tls" pod "09396946-8b1b-44e9-a0e6-b78ea69e6d6a" (UID: "09396946-8b1b-44e9-a0e6-b78ea69e6d6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:40:42.305165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.304863 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-metrics-certs" pod "09396946-8b1b-44e9-a0e6-b78ea69e6d6a" (UID: "09396946-8b1b-44e9-a0e6-b78ea69e6d6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:40:42.378322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.377683 5086 generic.go:356] "Generic (PLEG): container finished" podID="09396946-8b1b-44e9-a0e6-b78ea69e6d6a" containerID="1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783" exitCode=0 Nov 03 10:40:42.378322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.377883 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" event={"ID":"09396946-8b1b-44e9-a0e6-b78ea69e6d6a","Type":"ContainerDied","Data":"1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783"} Nov 03 10:40:42.378322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.377936 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" event={"ID":"09396946-8b1b-44e9-a0e6-b78ea69e6d6a","Type":"ContainerDied","Data":"53941a9247a256ef03a18d0bf78e52d9510c5edc01e8113e369be06e09904575"} Nov 03 10:40:42.378322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.377967 5086 scope.go:122] "RemoveContainer" containerID="1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783" Nov 03 10:40:42.378322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.378076 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k" Nov 03 10:40:42.403352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.402977 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:42.403352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.403011 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:42.403352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.403026 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6bqt\" (UniqueName: \"kubernetes.io/projected/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-kube-api-access-j6bqt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:42.403352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.403042 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09396946-8b1b-44e9-a0e6-b78ea69e6d6a-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:42.514228 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.514092 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k"] Nov 03 10:40:42.523345 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.521023 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-afe85416-7c48854c69-rxx2k"] Nov 03 10:40:42.534016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.533967 5086 scope.go:122] "RemoveContainer" containerID="1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783" Nov 03 10:40:42.534270 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:42.534241 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783\": container with ID starting with 1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783 not found: ID does not exist" containerID="1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783" Nov 03 10:40:42.534423 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.534278 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783"} err="failed to get container status \"1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783\": rpc error: code = NotFound desc = could not find container \"1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783\": container with ID starting with 1105b5aa42acf82c9835c543eda27b9d8682772af191bcc54489d1a859f6c783 not found: ID does not exist" Nov 03 10:40:42.938479 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.938446 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd"] Nov 03 10:40:42.938710 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.938687 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" containerID="cri-o://e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598" gracePeriod=30 Nov 03 10:40:42.978869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.978834 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4"] Nov 03 10:40:42.979095 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:42.979064 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" podUID="a5f26fb3-6e2f-490e-bc03-dd84358e3c34" containerName="maas-api" containerID="cri-o://936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63" gracePeriod=30 Nov 03 10:40:43.258011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.257984 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:43.319554 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.319525 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-kube-api-access-w5wk5\" (UniqueName: \"kubernetes.io/projected/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-kube-api-access-w5wk5\") pod \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " Nov 03 10:40:43.319725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.319711 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-metrics-certs\") pod \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " Nov 03 10:40:43.319766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.319739 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-openshift-service-ca\") pod \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " Nov 03 10:40:43.319803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.319778 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-maas-api-tls\") pod \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\" (UID: \"a5f26fb3-6e2f-490e-bc03-dd84358e3c34\") " Nov 03 10:40:43.320143 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.320114 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-openshift-service-ca" pod "a5f26fb3-6e2f-490e-bc03-dd84358e3c34" (UID: "a5f26fb3-6e2f-490e-bc03-dd84358e3c34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:40:43.321736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.321716 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-kube-api-access-w5wk5" pod "a5f26fb3-6e2f-490e-bc03-dd84358e3c34" (UID: "a5f26fb3-6e2f-490e-bc03-dd84358e3c34"). InnerVolumeSpecName "kube-api-access-w5wk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:43.321817 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.321801 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-maas-api-tls" pod "a5f26fb3-6e2f-490e-bc03-dd84358e3c34" (UID: "a5f26fb3-6e2f-490e-bc03-dd84358e3c34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:40:43.321923 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.321903 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-metrics-certs" pod "a5f26fb3-6e2f-490e-bc03-dd84358e3c34" (UID: "a5f26fb3-6e2f-490e-bc03-dd84358e3c34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:40:43.385675 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.385644 5086 generic.go:356] "Generic (PLEG): container finished" podID="a5f26fb3-6e2f-490e-bc03-dd84358e3c34" containerID="936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63" exitCode=0 Nov 03 10:40:43.386106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.385697 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" event={"ID":"a5f26fb3-6e2f-490e-bc03-dd84358e3c34","Type":"ContainerDied","Data":"936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63"} Nov 03 10:40:43.386106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.385732 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" event={"ID":"a5f26fb3-6e2f-490e-bc03-dd84358e3c34","Type":"ContainerDied","Data":"3206e263940d21af57882717788d9cc80bd54f2ca8a475002a2e2d7631f6dde6"} Nov 03 10:40:43.386106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.385752 5086 scope.go:122] "RemoveContainer" containerID="936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63" Nov 03 10:40:43.386106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.385697 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4" Nov 03 10:40:43.395600 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.395582 5086 scope.go:122] "RemoveContainer" containerID="936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63" Nov 03 10:40:43.395873 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:43.395856 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63\": container with ID starting with 936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63 not found: ID does not exist" containerID="936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63" Nov 03 10:40:43.395927 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.395885 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63"} err="failed to get container status \"936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63\": rpc error: code = NotFound desc = could not find container \"936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63\": container with ID starting with 936311cde350a7680f2e426c14d46d52e1f7ec6af0b8c373c42666aff977ba63 not found: ID does not exist" Nov 03 10:40:43.407911 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.407889 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4"] Nov 03 10:40:43.415860 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.415832 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19708dae-697d49db-ckmj4"] Nov 03 10:40:43.420790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.420764 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:43.420790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.420789 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:43.420923 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.420800 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:43.420923 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.420809 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5wk5\" (UniqueName: \"kubernetes.io/projected/a5f26fb3-6e2f-490e-bc03-dd84358e3c34-kube-api-access-w5wk5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:43.447802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.447768 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq"] Nov 03 10:40:43.448219 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:43.448035 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" containerID="cri-o://5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7" gracePeriod=30 Nov 03 10:40:44.170865 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:44.170830 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09396946-8b1b-44e9-a0e6-b78ea69e6d6a" path="/var/lib/kubelet/pods/09396946-8b1b-44e9-a0e6-b78ea69e6d6a/volumes" Nov 03 10:40:44.171244 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:44.171231 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f26fb3-6e2f-490e-bc03-dd84358e3c34" path="/var/lib/kubelet/pods/a5f26fb3-6e2f-490e-bc03-dd84358e3c34/volumes" Nov 03 10:40:46.272541 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.272508 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7"] Nov 03 10:40:46.272976 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.272962 5086 cpu_manager.go:414] "containerMap: removing container" podUID="fbe3241e-5fae-4156-aaa1-d514d65f9917" containerName="payload-processing" Nov 03 10:40:46.273036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.272977 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe3241e-5fae-4156-aaa1-d514d65f9917" containerName="payload-processing" Nov 03 10:40:46.273036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.272985 5086 cpu_manager.go:414] "containerMap: removing container" podUID="a5f26fb3-6e2f-490e-bc03-dd84358e3c34" containerName="maas-api" Nov 03 10:40:46.273036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.272990 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f26fb3-6e2f-490e-bc03-dd84358e3c34" containerName="maas-api" Nov 03 10:40:46.273036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273022 5086 cpu_manager.go:414] "containerMap: removing container" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" Nov 03 10:40:46.273036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273028 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" Nov 03 10:40:46.273036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273038 5086 cpu_manager.go:414] "containerMap: removing container" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273043 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273050 5086 cpu_manager.go:414] "containerMap: removing container" podUID="adb92568-0036-4ee3-b70a-6b096f117fa4" containerName="maas-api" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273054 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb92568-0036-4ee3-b70a-6b096f117fa4" containerName="maas-api" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273061 5086 cpu_manager.go:414] "containerMap: removing container" podUID="09396946-8b1b-44e9-a0e6-b78ea69e6d6a" containerName="maas-api" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273066 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09396946-8b1b-44e9-a0e6-b78ea69e6d6a" containerName="maas-api" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273146 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273154 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb0ba09-173d-44e7-8215-ce4950c33a41" containerName="istio-proxy" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273162 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb92568-0036-4ee3-b70a-6b096f117fa4" containerName="maas-api" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273169 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="09396946-8b1b-44e9-a0e6-b78ea69e6d6a" containerName="maas-api" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273178 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe3241e-5fae-4156-aaa1-d514d65f9917" containerName="payload-processing" Nov 03 10:40:46.273204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.273184 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f26fb3-6e2f-490e-bc03-dd84358e3c34" containerName="maas-api" Nov 03 10:40:46.277685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.277668 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:46.286043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.286018 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-010232\"" Nov 03 10:40:46.286512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.286490 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7"] Nov 03 10:40:46.336974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.336942 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc"] Nov 03 10:40:46.340641 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.340624 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:46.348869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.348839 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-010232-dockercfg-9hxq6\"" Nov 03 10:40:46.353678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.353654 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc"] Nov 03 10:40:46.354952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.354922 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7\" (UID: \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:46.355184 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.355161 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7jx\" (UniqueName: \"kubernetes.io/projected/6e657caa-0b2b-4318-ac6f-1d1e811992d5-kube-api-access-8v7jx\") pod \"payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc\" (UID: \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:46.355350 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.355331 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e657caa-0b2b-4318-ac6f-1d1e811992d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc\" (UID: \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:46.355496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.355478 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq46c\" (UniqueName: \"kubernetes.io/projected/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-kube-api-access-gq46c\") pod \"payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7\" (UID: \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:46.429704 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.429642 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" podUID="ae9ccf93-9d99-4aa1-a4fe-642731db5b00" containerName="payload-processing" containerID="cri-o://dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83" gracePeriod=25 Nov 03 10:40:46.456721 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.456689 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7jx\" (UniqueName: \"kubernetes.io/projected/6e657caa-0b2b-4318-ac6f-1d1e811992d5-kube-api-access-8v7jx\") pod \"payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc\" (UID: \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:46.456826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.456738 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e657caa-0b2b-4318-ac6f-1d1e811992d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc\" (UID: \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:46.456826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.456766 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq46c\" (UniqueName: \"kubernetes.io/projected/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-kube-api-access-gq46c\") pod \"payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7\" (UID: \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:46.456918 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.456879 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7\" (UID: \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:46.457472 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.457450 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e657caa-0b2b-4318-ac6f-1d1e811992d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc\" (UID: \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:46.458125 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.458106 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7\" (UID: \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:46.467612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.467591 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7jx\" (UniqueName: \"kubernetes.io/projected/6e657caa-0b2b-4318-ac6f-1d1e811992d5-kube-api-access-8v7jx\") pod \"payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc\" (UID: \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:46.467943 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.467924 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq46c\" (UniqueName: \"kubernetes.io/projected/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-kube-api-access-gq46c\") pod \"payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7\" (UID: \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:46.561923 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.561884 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5"] Nov 03 10:40:46.568708 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.568680 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.584316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.583847 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 10:40:46.584316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.584007 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 10:40:46.584316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.584081 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-010232-serving-cert\"" Nov 03 10:40:46.586911 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.586646 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 10:40:46.589686 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.587445 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l8bvs\"" Nov 03 10:40:46.589686 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.588327 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5"] Nov 03 10:40:46.594766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.594561 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:46.664674 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.664642 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:46.683678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.667655 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.683678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.667730 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-metrics-certs\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.683678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.667765 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.683678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.667875 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs6fq\" (UniqueName: \"kubernetes.io/projected/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-kube-api-access-cs6fq\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.765952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.765923 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:46.769249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.769199 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs6fq\" (UniqueName: \"kubernetes.io/projected/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-kube-api-access-cs6fq\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.769399 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.769353 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.769467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.769413 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-metrics-certs\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.769467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.769449 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.770996 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.770358 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.774905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.774882 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.777513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.777492 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-metrics-certs\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.779424 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.779399 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs6fq\" (UniqueName: \"kubernetes.io/projected/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-kube-api-access-cs6fq\") pod \"maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.870097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.870032 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-plugins-config-volume\") pod \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\" (UID: \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\") " Nov 03 10:40:46.870341 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.870113 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-kube-api-access-69879\" (UniqueName: \"kubernetes.io/projected/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-kube-api-access-69879\") pod \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\" (UID: \"ae9ccf93-9d99-4aa1-a4fe-642731db5b00\") " Nov 03 10:40:46.870878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.870845 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-plugins-config-volume" pod "ae9ccf93-9d99-4aa1-a4fe-642731db5b00" (UID: "ae9ccf93-9d99-4aa1-a4fe-642731db5b00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:40:46.872484 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.872452 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-kube-api-access-69879" pod "ae9ccf93-9d99-4aa1-a4fe-642731db5b00" (UID: "ae9ccf93-9d99-4aa1-a4fe-642731db5b00"). InnerVolumeSpecName "kube-api-access-69879". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:46.901647 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.899972 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:46.902087 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.902057 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7"] Nov 03 10:40:46.902902 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:46.902865 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ef7069_f15c_4d0e_89d3_2724ff2a0fb2.slice/crio-eeb09d15304a0875b5f1848c7e59754a55565fd26912516440ecfcbe9f080756 WatchSource:0}: Error finding container eeb09d15304a0875b5f1848c7e59754a55565fd26912516440ecfcbe9f080756: Status 404 returned error can't find the container with id eeb09d15304a0875b5f1848c7e59754a55565fd26912516440ecfcbe9f080756 Nov 03 10:40:46.971228 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.971159 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:46.971228 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.971198 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69879\" (UniqueName: \"kubernetes.io/projected/ae9ccf93-9d99-4aa1-a4fe-642731db5b00-kube-api-access-69879\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:46.984654 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:46.984606 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc"] Nov 03 10:40:46.986149 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:46.986096 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e657caa_0b2b_4318_ac6f_1d1e811992d5.slice/crio-24b61e0ca1cd5dcd43f20c9ad338af4c80c16041b4da1934d4dbd9a1151365ce WatchSource:0}: Error finding container 24b61e0ca1cd5dcd43f20c9ad338af4c80c16041b4da1934d4dbd9a1151365ce: Status 404 returned error can't find the container with id 24b61e0ca1cd5dcd43f20c9ad338af4c80c16041b4da1934d4dbd9a1151365ce Nov 03 10:40:47.122720 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.122640 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng"] Nov 03 10:40:47.123163 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.123143 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ae9ccf93-9d99-4aa1-a4fe-642731db5b00" containerName="payload-processing" Nov 03 10:40:47.123163 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.123161 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9ccf93-9d99-4aa1-a4fe-642731db5b00" containerName="payload-processing" Nov 03 10:40:47.123343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.123258 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9ccf93-9d99-4aa1-a4fe-642731db5b00" containerName="payload-processing" Nov 03 10:40:47.127981 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.127961 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:40:47.133873 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.133810 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng"] Nov 03 10:40:47.137981 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.137960 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d5d3c3\"" Nov 03 10:40:47.168620 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.168595 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-d5d3c3-dockercfg-kql6v\"" Nov 03 10:40:47.172999 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.172974 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52b7m\" (UniqueName: \"kubernetes.io/projected/7124d01f-392b-490a-9688-3a4e3b9f50f1-kube-api-access-52b7m\") pod \"payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng\" (UID: \"7124d01f-392b-490a-9688-3a4e3b9f50f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:40:47.173124 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.173023 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7124d01f-392b-490a-9688-3a4e3b9f50f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng\" (UID: \"7124d01f-392b-490a-9688-3a4e3b9f50f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:40:47.177884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.177863 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5"] Nov 03 10:40:47.179708 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:47.179683 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fab6dfc_dfe0_4ec8_bd5d_e59953b0290d.slice/crio-0d1cd00d88407d905667d89d637a14a477c7922f46fb1aa4ebd54c49cee4f973 WatchSource:0}: Error finding container 0d1cd00d88407d905667d89d637a14a477c7922f46fb1aa4ebd54c49cee4f973: Status 404 returned error can't find the container with id 0d1cd00d88407d905667d89d637a14a477c7922f46fb1aa4ebd54c49cee4f973 Nov 03 10:40:47.274516 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.274478 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7124d01f-392b-490a-9688-3a4e3b9f50f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng\" (UID: \"7124d01f-392b-490a-9688-3a4e3b9f50f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:40:47.275032 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.274689 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52b7m\" (UniqueName: \"kubernetes.io/projected/7124d01f-392b-490a-9688-3a4e3b9f50f1-kube-api-access-52b7m\") pod \"payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng\" (UID: \"7124d01f-392b-490a-9688-3a4e3b9f50f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:40:47.275367 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.275330 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7124d01f-392b-490a-9688-3a4e3b9f50f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng\" (UID: \"7124d01f-392b-490a-9688-3a4e3b9f50f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:40:47.282312 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.282275 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52b7m\" (UniqueName: \"kubernetes.io/projected/7124d01f-392b-490a-9688-3a4e3b9f50f1-kube-api-access-52b7m\") pod \"payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng\" (UID: \"7124d01f-392b-490a-9688-3a4e3b9f50f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:40:47.413427 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.413326 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" event={"ID":"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2","Type":"ContainerStarted","Data":"b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de"} Nov 03 10:40:47.413427 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.413395 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" event={"ID":"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2","Type":"ContainerStarted","Data":"eeb09d15304a0875b5f1848c7e59754a55565fd26912516440ecfcbe9f080756"} Nov 03 10:40:47.413634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.413525 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:47.414587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.414561 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" event={"ID":"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d","Type":"ContainerStarted","Data":"0d1cd00d88407d905667d89d637a14a477c7922f46fb1aa4ebd54c49cee4f973"} Nov 03 10:40:47.416113 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.416092 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" event={"ID":"6e657caa-0b2b-4318-ac6f-1d1e811992d5","Type":"ContainerStarted","Data":"b63b714cc03530e21f7322f473c9dc353f74f67110bd862a61f068d13049c148"} Nov 03 10:40:47.416207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.416138 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:47.416207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.416162 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" event={"ID":"6e657caa-0b2b-4318-ac6f-1d1e811992d5","Type":"ContainerStarted","Data":"24b61e0ca1cd5dcd43f20c9ad338af4c80c16041b4da1934d4dbd9a1151365ce"} Nov 03 10:40:47.417488 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.417464 5086 generic.go:356] "Generic (PLEG): container finished" podID="ae9ccf93-9d99-4aa1-a4fe-642731db5b00" containerID="dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83" exitCode=0 Nov 03 10:40:47.417564 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.417505 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" Nov 03 10:40:47.417564 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.417541 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" event={"ID":"ae9ccf93-9d99-4aa1-a4fe-642731db5b00","Type":"ContainerDied","Data":"dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83"} Nov 03 10:40:47.417684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.417587 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64" event={"ID":"ae9ccf93-9d99-4aa1-a4fe-642731db5b00","Type":"ContainerDied","Data":"ce6e9f8c62e7665ffcf91dd59070ecf2ee92a277c40d58ef0a53a4fe7495e0a8"} Nov 03 10:40:47.417684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.417612 5086 scope.go:122] "RemoveContainer" containerID="dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83" Nov 03 10:40:47.432458 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.432413 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" podStartSLOduration=1.432400246 podStartE2EDuration="1.432400246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:47.428620562 +0000 UTC m=+725.413491845" watchObservedRunningTime="2026-11-03 10:40:47.432400246 +0000 UTC m=+725.417271528" Nov 03 10:40:47.434490 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.434175 5086 scope.go:122] "RemoveContainer" containerID="dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83" Nov 03 10:40:47.434586 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:47.434541 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83\": container with ID starting with dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83 not found: ID does not exist" containerID="dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83" Nov 03 10:40:47.434679 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.434579 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83"} err="failed to get container status \"dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83\": rpc error: code = NotFound desc = could not find container \"dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83\": container with ID starting with dc905d6844bce0c38dba0c7b97141682301a9e47687f8d45a9d696f182330c83 not found: ID does not exist" Nov 03 10:40:47.446149 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.446102 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" podStartSLOduration=1.4460873410000001 podStartE2EDuration="1.446087341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:47.444225283 +0000 UTC m=+725.429096566" watchObservedRunningTime="2026-11-03 10:40:47.446087341 +0000 UTC m=+725.430958623" Nov 03 10:40:47.449034 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.449004 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:40:47.458744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.458668 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64"] Nov 03 10:40:47.460862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.460838 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-afe85416-77455b99f-ffk64"] Nov 03 10:40:47.731727 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:47.731693 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng"] Nov 03 10:40:47.733085 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:47.733061 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7124d01f_392b_490a_9688_3a4e3b9f50f1.slice/crio-798f870da86c9fe8c70c2fd314ee5d5a2c73b039b6edf920c712c8b2b8de905c WatchSource:0}: Error finding container 798f870da86c9fe8c70c2fd314ee5d5a2c73b039b6edf920c712c8b2b8de905c: Status 404 returned error can't find the container with id 798f870da86c9fe8c70c2fd314ee5d5a2c73b039b6edf920c712c8b2b8de905c Nov 03 10:40:48.173442 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.173410 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae9ccf93-9d99-4aa1-a4fe-642731db5b00" path="/var/lib/kubelet/pods/ae9ccf93-9d99-4aa1-a4fe-642731db5b00/volumes" Nov 03 10:40:48.199655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.199633 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:48.283020 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.282944 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-podinfo\") pod \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " Nov 03 10:40:48.283020 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.282984 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-credential-socket\") pod \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " Nov 03 10:40:48.283020 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283010 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-certs\") pod \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " Nov 03 10:40:48.283700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283033 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-envoy\") pod \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " Nov 03 10:40:48.283700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283171 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-kube-api-access-kdx7r\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-kube-api-access-kdx7r\") pod \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " Nov 03 10:40:48.283700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283197 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-socket\") pod \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " Nov 03 10:40:48.283700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283235 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-token\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-token\") pod \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " Nov 03 10:40:48.283700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283238 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-credential-socket" pod "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" (UID: "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.283700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283278 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istiod-ca-cert\") pod \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " Nov 03 10:40:48.283700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283336 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-data\") pod \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\" (UID: \"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1\") " Nov 03 10:40:48.283700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283423 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-certs" pod "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" (UID: "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.283700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283498 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-socket" pod "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" (UID: "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.284201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283761 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.284201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283781 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.284201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.283796 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.284201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.284025 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istiod-ca-cert" pod "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" (UID: "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:40:48.284383 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.284246 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-data" pod "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" (UID: "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.285745 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.285726 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-envoy" pod "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" (UID: "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.285877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.285857 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-podinfo" pod "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" (UID: "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:40:48.285916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.285882 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-kube-api-access-kdx7r" pod "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" (UID: "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1"). InnerVolumeSpecName "kube-api-access-kdx7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:48.286045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.286030 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-token" pod "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" (UID: "fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:48.384916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.384876 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdx7r\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-kube-api-access-kdx7r\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.384916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.384916 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.385120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.384931 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.385120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.384945 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.385120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.384960 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.385120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.384975 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.428208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.428166 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" event={"ID":"7124d01f-392b-490a-9688-3a4e3b9f50f1","Type":"ContainerStarted","Data":"71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f"} Nov 03 10:40:48.428406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.428237 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" event={"ID":"7124d01f-392b-490a-9688-3a4e3b9f50f1","Type":"ContainerStarted","Data":"798f870da86c9fe8c70c2fd314ee5d5a2c73b039b6edf920c712c8b2b8de905c"} Nov 03 10:40:48.428497 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.428423 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:40:48.429864 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.429829 5086 generic.go:356] "Generic (PLEG): container finished" podID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerID="e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598" exitCode=0 Nov 03 10:40:48.429980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.429877 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" Nov 03 10:40:48.429980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.429883 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" event={"ID":"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1","Type":"ContainerDied","Data":"e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598"} Nov 03 10:40:48.429980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.429929 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd" event={"ID":"fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1","Type":"ContainerDied","Data":"c9159a6cfa1eb40c1f04bfd2e8af0046d9f26929a9d66aa2fe62ab00d2d43a22"} Nov 03 10:40:48.429980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.429958 5086 scope.go:122] "RemoveContainer" containerID="e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598" Nov 03 10:40:48.431845 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.431816 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" event={"ID":"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d","Type":"ContainerStarted","Data":"813df2b8fe2109da4b91e3fe85e6eae71ec75bfc7884a16af7566475d70fde9d"} Nov 03 10:40:48.432065 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.432038 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:48.440096 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.439842 5086 scope.go:122] "RemoveContainer" containerID="c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72" Nov 03 10:40:48.444838 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.444795 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" podStartSLOduration=1.444784739 podStartE2EDuration="1.444784739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:48.443531823 +0000 UTC m=+726.428403106" watchObservedRunningTime="2026-11-03 10:40:48.444784739 +0000 UTC m=+726.429656021" Nov 03 10:40:48.460639 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.460620 5086 scope.go:122] "RemoveContainer" containerID="e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598" Nov 03 10:40:48.461000 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:48.460980 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598\": container with ID starting with e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598 not found: ID does not exist" containerID="e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598" Nov 03 10:40:48.461097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.461006 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598"} err="failed to get container status \"e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598\": rpc error: code = NotFound desc = could not find container \"e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598\": container with ID starting with e0e8dbca1a16496e1410d7f3e9485a079d634202366a4cf43ba9cd37ef48d598 not found: ID does not exist" Nov 03 10:40:48.461097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.461025 5086 scope.go:122] "RemoveContainer" containerID="c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72" Nov 03 10:40:48.461286 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:48.461265 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72\": container with ID starting with c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72 not found: ID does not exist" containerID="c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72" Nov 03 10:40:48.461373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.461324 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72"} err="failed to get container status \"c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72\": rpc error: code = NotFound desc = could not find container \"c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72\": container with ID starting with c96e46257583f7109fbdaa5dac5affce24d636ee983ac8cc3ef6ee3ad6a2be72 not found: ID does not exist" Nov 03 10:40:48.466151 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.466108 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" podStartSLOduration=1.9708147550000001 podStartE2EDuration="2.46609539s" totalImagesPullingTime="495.280635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:48.463205571 +0000 UTC m=+726.448076856" watchObservedRunningTime="2026-11-03 10:40:48.46609539 +0000 UTC m=+726.450966673" Nov 03 10:40:48.489168 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.489139 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd"] Nov 03 10:40:48.491181 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.491158 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-afe85416-openshift-default-75b5b9d5c7-pqzdd"] Nov 03 10:40:48.549545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.549454 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq"] Nov 03 10:40:48.550036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.550016 5086 cpu_manager.go:414] "containerMap: removing container" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" Nov 03 10:40:48.550036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.550033 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" Nov 03 10:40:48.550210 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.550162 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" Nov 03 10:40:48.553723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.553698 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.571334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.565427 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-d5d3c3-serving-cert\"" Nov 03 10:40:48.571334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.566517 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq"] Nov 03 10:40:48.587881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.587359 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89j5\" (UniqueName: \"kubernetes.io/projected/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-kube-api-access-p89j5\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.587881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.587444 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.587881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.587568 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-metrics-certs\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.587881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.587754 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.688657 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.688618 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89j5\" (UniqueName: \"kubernetes.io/projected/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-kube-api-access-p89j5\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.688828 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.688669 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.688828 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.688729 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-metrics-certs\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.688828 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.688770 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.689588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.689562 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.691781 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.691750 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.691897 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.691751 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-metrics-certs\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.696953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.696924 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89j5\" (UniqueName: \"kubernetes.io/projected/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-kube-api-access-p89j5\") pod \"maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.764312 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.764274 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:48.789857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.789827 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-socket\") pod \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " Nov 03 10:40:48.790192 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.789892 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istiod-ca-cert\") pod \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " Nov 03 10:40:48.790192 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.789945 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-data\") pod \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " Nov 03 10:40:48.790192 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790016 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-kube-api-access-xj5mv\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-kube-api-access-xj5mv\") pod \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " Nov 03 10:40:48.790192 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790128 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-podinfo\") pod \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " Nov 03 10:40:48.790192 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790191 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-certs\") pod \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " Nov 03 10:40:48.790558 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790216 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-token\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-token\") pod \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " Nov 03 10:40:48.790558 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790220 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-socket" pod "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" (UID: "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.790558 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790244 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istiod-ca-cert" pod "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" (UID: "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:40:48.790558 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790246 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-credential-socket\") pod \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " Nov 03 10:40:48.790558 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790317 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-envoy\") pod \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\" (UID: \"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29\") " Nov 03 10:40:48.790751 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790578 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-data" pod "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" (UID: "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.790751 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790564 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-credential-socket" pod "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" (UID: "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.790751 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790602 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-certs" pod "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" (UID: "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.790897 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790871 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.791009 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790899 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.791009 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790913 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.791009 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790924 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.791009 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.790936 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.792988 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.792939 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-podinfo" pod "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" (UID: "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:40:48.792988 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.792951 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-envoy" pod "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" (UID: "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:40:48.793350 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.793283 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-token" pod "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" (UID: "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:48.793932 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.793912 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-kube-api-access-xj5mv" pod "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" (UID: "2f03a7ba-dcc4-4f79-b366-f7ce6f878f29"). InnerVolumeSpecName "kube-api-access-xj5mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:40:48.882541 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.882508 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:48.892149 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.892121 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.892149 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.892144 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.892149 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.892156 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:48.892422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:48.892166 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj5mv\" (UniqueName: \"kubernetes.io/projected/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29-kube-api-access-xj5mv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:40:49.163052 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.163022 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq"] Nov 03 10:40:49.164848 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:49.164821 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54b5afc_e80a_4c14_aafd_8ff9ad7da046.slice/crio-8d2fe6aeac7f0acf2b0794e2a2092a85429f612ec6cfc450bc86247dba833be9 WatchSource:0}: Error finding container 8d2fe6aeac7f0acf2b0794e2a2092a85429f612ec6cfc450bc86247dba833be9: Status 404 returned error can't find the container with id 8d2fe6aeac7f0acf2b0794e2a2092a85429f612ec6cfc450bc86247dba833be9 Nov 03 10:40:49.326612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.326558 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms"] Nov 03 10:40:49.327408 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.327384 5086 cpu_manager.go:414] "containerMap: removing container" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" Nov 03 10:40:49.327529 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.327409 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" Nov 03 10:40:49.327529 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.327426 5086 cpu_manager.go:414] "containerMap: removing container" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" Nov 03 10:40:49.327529 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.327433 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" Nov 03 10:40:49.327529 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.327484 5086 cpu_manager.go:414] "containerMap: removing container" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" Nov 03 10:40:49.327529 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.327492 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" Nov 03 10:40:49.327763 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.327624 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" containerName="istio-proxy" Nov 03 10:40:49.327763 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.327638 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" Nov 03 10:40:49.332206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.332185 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:40:49.339663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.339634 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms"] Nov 03 10:40:49.341025 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.340476 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e78c33-dockercfg-qmk5x\"" Nov 03 10:40:49.359124 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.359090 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb"] Nov 03 10:40:49.359682 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.359662 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerName="istio-proxy" Nov 03 10:40:49.362996 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.362976 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:40:49.370078 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.370055 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb"] Nov 03 10:40:49.398321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.398273 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcbn6\" (UniqueName: \"kubernetes.io/projected/0f294516-cbba-4b13-87f4-e8a9815f82c8-kube-api-access-gcbn6\") pod \"payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb\" (UID: \"0f294516-cbba-4b13-87f4-e8a9815f82c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:40:49.398496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.398334 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a59fd-e1ee-402e-91f0-ff267e768e24-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms\" (UID: \"692a59fd-e1ee-402e-91f0-ff267e768e24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:40:49.398545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.398490 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qbt9\" (UniqueName: \"kubernetes.io/projected/692a59fd-e1ee-402e-91f0-ff267e768e24-kube-api-access-9qbt9\") pod \"payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms\" (UID: \"692a59fd-e1ee-402e-91f0-ff267e768e24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:40:49.398631 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.398611 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f294516-cbba-4b13-87f4-e8a9815f82c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb\" (UID: \"0f294516-cbba-4b13-87f4-e8a9815f82c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:40:49.429869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.429776 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e78c33\"" Nov 03 10:40:49.459775 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.459550 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" event={"ID":"d54b5afc-e80a-4c14-aafd-8ff9ad7da046","Type":"ContainerStarted","Data":"8d2fe6aeac7f0acf2b0794e2a2092a85429f612ec6cfc450bc86247dba833be9"} Nov 03 10:40:49.463514 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.463486 5086 generic.go:356] "Generic (PLEG): container finished" podID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" containerID="5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7" exitCode=0 Nov 03 10:40:49.464469 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.464450 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" Nov 03 10:40:49.466489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.466461 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" event={"ID":"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29","Type":"ContainerDied","Data":"5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7"} Nov 03 10:40:49.466849 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.466828 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq" event={"ID":"2f03a7ba-dcc4-4f79-b366-f7ce6f878f29","Type":"ContainerDied","Data":"c1efdcd860c0149ebdfe9852cd3b911f6944b66251b3ed39fdfd14622f3f7f63"} Nov 03 10:40:49.467034 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.466982 5086 scope.go:122] "RemoveContainer" containerID="5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7" Nov 03 10:40:49.485351 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.485327 5086 scope.go:122] "RemoveContainer" containerID="0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a" Nov 03 10:40:49.494557 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.494525 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq"] Nov 03 10:40:49.496244 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.496224 5086 scope.go:122] "RemoveContainer" containerID="5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7" Nov 03 10:40:49.496674 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:49.496643 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7\": container with ID starting with 5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7 not found: ID does not exist" containerID="5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7" Nov 03 10:40:49.496754 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.496682 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7"} err="failed to get container status \"5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7\": rpc error: code = NotFound desc = could not find container \"5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7\": container with ID starting with 5753a9e758f54c96d93c5c73d1926b8e1f7094113523f6c20b0fbedc61cd85e7 not found: ID does not exist" Nov 03 10:40:49.496754 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.496720 5086 scope.go:122] "RemoveContainer" containerID="0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a" Nov 03 10:40:49.497064 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:49.497042 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a\": container with ID starting with 0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a not found: ID does not exist" containerID="0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a" Nov 03 10:40:49.497145 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.497073 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a"} err="failed to get container status \"0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a\": rpc error: code = NotFound desc = could not find container \"0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a\": container with ID starting with 0de586b78ab44225dc98148d12dbfbe3fadd0c41e6acd2bdf8820f3d84cdad8a not found: ID does not exist" Nov 03 10:40:49.497907 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.497888 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-19708dae-openshift-default-85bbdddd47-5rldq"] Nov 03 10:40:49.499602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.499581 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f294516-cbba-4b13-87f4-e8a9815f82c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb\" (UID: \"0f294516-cbba-4b13-87f4-e8a9815f82c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:40:49.499684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.499640 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcbn6\" (UniqueName: \"kubernetes.io/projected/0f294516-cbba-4b13-87f4-e8a9815f82c8-kube-api-access-gcbn6\") pod \"payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb\" (UID: \"0f294516-cbba-4b13-87f4-e8a9815f82c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:40:49.499737 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.499679 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a59fd-e1ee-402e-91f0-ff267e768e24-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms\" (UID: \"692a59fd-e1ee-402e-91f0-ff267e768e24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:40:49.500156 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.499822 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qbt9\" (UniqueName: \"kubernetes.io/projected/692a59fd-e1ee-402e-91f0-ff267e768e24-kube-api-access-9qbt9\") pod \"payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms\" (UID: \"692a59fd-e1ee-402e-91f0-ff267e768e24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:40:49.500791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.500773 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f294516-cbba-4b13-87f4-e8a9815f82c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb\" (UID: \"0f294516-cbba-4b13-87f4-e8a9815f82c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:40:49.501014 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.500998 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a59fd-e1ee-402e-91f0-ff267e768e24-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms\" (UID: \"692a59fd-e1ee-402e-91f0-ff267e768e24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:40:49.509802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.509778 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qbt9\" (UniqueName: \"kubernetes.io/projected/692a59fd-e1ee-402e-91f0-ff267e768e24-kube-api-access-9qbt9\") pod \"payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms\" (UID: \"692a59fd-e1ee-402e-91f0-ff267e768e24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:40:49.510320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.510268 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcbn6\" (UniqueName: \"kubernetes.io/projected/0f294516-cbba-4b13-87f4-e8a9815f82c8-kube-api-access-gcbn6\") pod \"payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb\" (UID: \"0f294516-cbba-4b13-87f4-e8a9815f82c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:40:49.652083 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.652055 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:40:49.683548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.683471 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:40:49.937447 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:49.937399 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms"] Nov 03 10:40:49.939770 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:49.939744 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692a59fd_e1ee_402e_91f0_ff267e768e24.slice/crio-8b405c8494a3a678ad16ecb4bd4689c4f80a459f1cbf6377e96d9c968840f590 WatchSource:0}: Error finding container 8b405c8494a3a678ad16ecb4bd4689c4f80a459f1cbf6377e96d9c968840f590: Status 404 returned error can't find the container with id 8b405c8494a3a678ad16ecb4bd4689c4f80a459f1cbf6377e96d9c968840f590 Nov 03 10:40:50.003943 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.003867 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb"] Nov 03 10:40:50.008463 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:50.008437 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f294516_cbba_4b13_87f4_e8a9815f82c8.slice/crio-4a2236bcab6ecf71949f17e6e344f184d5d292c001f7063efc5dddb5dc0b0e1e WatchSource:0}: Error finding container 4a2236bcab6ecf71949f17e6e344f184d5d292c001f7063efc5dddb5dc0b0e1e: Status 404 returned error can't find the container with id 4a2236bcab6ecf71949f17e6e344f184d5d292c001f7063efc5dddb5dc0b0e1e Nov 03 10:40:50.172891 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.172858 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f03a7ba-dcc4-4f79-b366-f7ce6f878f29" path="/var/lib/kubelet/pods/2f03a7ba-dcc4-4f79-b366-f7ce6f878f29/volumes" Nov 03 10:40:50.173497 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.173478 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1" path="/var/lib/kubelet/pods/fdda61d4-b6ba-47f3-a3a9-9fe9ead6bfa1/volumes" Nov 03 10:40:50.472024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.471977 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" event={"ID":"692a59fd-e1ee-402e-91f0-ff267e768e24","Type":"ContainerStarted","Data":"1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513"} Nov 03 10:40:50.472512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.472047 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" event={"ID":"692a59fd-e1ee-402e-91f0-ff267e768e24","Type":"ContainerStarted","Data":"8b405c8494a3a678ad16ecb4bd4689c4f80a459f1cbf6377e96d9c968840f590"} Nov 03 10:40:50.472512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.472180 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:40:50.474998 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.474972 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" event={"ID":"d54b5afc-e80a-4c14-aafd-8ff9ad7da046","Type":"ContainerStarted","Data":"be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33"} Nov 03 10:40:50.475129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.475112 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:50.476696 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.476673 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" event={"ID":"0f294516-cbba-4b13-87f4-e8a9815f82c8","Type":"ContainerStarted","Data":"d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204"} Nov 03 10:40:50.476811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.476711 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" event={"ID":"0f294516-cbba-4b13-87f4-e8a9815f82c8","Type":"ContainerStarted","Data":"4a2236bcab6ecf71949f17e6e344f184d5d292c001f7063efc5dddb5dc0b0e1e"} Nov 03 10:40:50.476811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.476796 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:40:50.487753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.487706 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" podStartSLOduration=1.487691186 podStartE2EDuration="1.487691186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:50.486436616 +0000 UTC m=+728.471307900" watchObservedRunningTime="2026-11-03 10:40:50.487691186 +0000 UTC m=+728.472562470" Nov 03 10:40:50.503858 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.503811 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" podStartSLOduration=2.110812659 podStartE2EDuration="2.50379681s" totalImagesPullingTime="392.984151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:50.50198488 +0000 UTC m=+728.486856175" watchObservedRunningTime="2026-11-03 10:40:50.50379681 +0000 UTC m=+728.488668093" Nov 03 10:40:50.524247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:50.524190 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" podStartSLOduration=1.5241738470000001 podStartE2EDuration="1.524173847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:50.523239717 +0000 UTC m=+728.508111002" watchObservedRunningTime="2026-11-03 10:40:50.524173847 +0000 UTC m=+728.509045131" Nov 03 10:40:51.061854 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.061813 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9"] Nov 03 10:40:51.067370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.067338 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.091633 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.091582 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e78c33-serving-cert\"" Nov 03 10:40:51.096878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.096847 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9"] Nov 03 10:40:51.117941 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.117901 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-metrics-certs\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.118109 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.117975 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmmj\" (UniqueName: \"kubernetes.io/projected/955750bc-952d-4b29-b6e8-a4c119428fba-kube-api-access-prmmj\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.118109 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.118036 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955750bc-952d-4b29-b6e8-a4c119428fba-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.118272 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.118144 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.219218 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.219183 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-metrics-certs\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.219457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.219346 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmmj\" (UniqueName: \"kubernetes.io/projected/955750bc-952d-4b29-b6e8-a4c119428fba-kube-api-access-prmmj\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.219457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.219419 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955750bc-952d-4b29-b6e8-a4c119428fba-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.219725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.219701 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.220607 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.220579 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955750bc-952d-4b29-b6e8-a4c119428fba-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.222280 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.222246 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-metrics-certs\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.222393 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.222333 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.227687 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.227657 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmmj\" (UniqueName: \"kubernetes.io/projected/955750bc-952d-4b29-b6e8-a4c119428fba-kube-api-access-prmmj\") pod \"maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.409967 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.409927 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:51.710013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:51.709981 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9"] Nov 03 10:40:51.712867 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:40:51.712829 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955750bc_952d_4b29_b6e8_a4c119428fba.slice/crio-cc3d7469b83f729160e11c7636453b30f7068119d19045919e65646562debd12 WatchSource:0}: Error finding container cc3d7469b83f729160e11c7636453b30f7068119d19045919e65646562debd12: Status 404 returned error can't find the container with id cc3d7469b83f729160e11c7636453b30f7068119d19045919e65646562debd12 Nov 03 10:40:52.494249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:52.494155 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" event={"ID":"955750bc-952d-4b29-b6e8-a4c119428fba","Type":"ContainerStarted","Data":"6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e"} Nov 03 10:40:52.494249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:52.494220 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" event={"ID":"955750bc-952d-4b29-b6e8-a4c119428fba","Type":"ContainerStarted","Data":"cc3d7469b83f729160e11c7636453b30f7068119d19045919e65646562debd12"} Nov 03 10:40:52.494249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:52.494249 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:52.512476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:52.512424 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" podStartSLOduration=1.105028868 podStartE2EDuration="1.512409839s" totalImagesPullingTime="407.380971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:52.509971992 +0000 UTC m=+730.494843277" watchObservedRunningTime="2026-11-03 10:40:52.512409839 +0000 UTC m=+730.497281121" Nov 03 10:40:54.275352 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:40:54.275312 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:40:54.471599 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:54.471568 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 10:40:56.491528 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:56.491495 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 10:40:58.435082 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:58.435053 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 10:40:58.435484 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:58.435123 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 10:40:58.506377 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:58.506350 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 10:40:59.469936 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:40:59.469906 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 10:41:01.285341 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.284587 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r"] Nov 03 10:41:01.290716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.289638 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.307440 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.307402 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f78621-serving-cert\"" Nov 03 10:41:01.312818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.312789 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r"] Nov 03 10:41:01.433385 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.433348 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-metrics-certs\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.433693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.433656 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3318e5-f88a-40e6-9482-55a0e47c162e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.433810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.433718 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.433810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.433775 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7jt\" (UniqueName: \"kubernetes.io/projected/ac3318e5-f88a-40e6-9482-55a0e47c162e-kube-api-access-nr7jt\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.490048 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.490019 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 10:41:01.490901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.490877 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 10:41:01.534770 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.534729 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-metrics-certs\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.535007 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.534830 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3318e5-f88a-40e6-9482-55a0e47c162e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.535007 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.534876 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.535007 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.534893 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7jt\" (UniqueName: \"kubernetes.io/projected/ac3318e5-f88a-40e6-9482-55a0e47c162e-kube-api-access-nr7jt\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.535980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.535911 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3318e5-f88a-40e6-9482-55a0e47c162e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.537739 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.537715 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.538279 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.538252 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-metrics-certs\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.567597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.567568 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7jt\" (UniqueName: \"kubernetes.io/projected/ac3318e5-f88a-40e6-9482-55a0e47c162e-kube-api-access-nr7jt\") pod \"maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.619658 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.619629 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:01.911953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:01.911924 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r"] Nov 03 10:41:01.914178 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:41:01.914126 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3318e5_f88a_40e6_9482_55a0e47c162e.slice/crio-74a4e34f428da8a2ff556d8dfd1f97fee823168f0f90e2b9904219b1c391caa7 WatchSource:0}: Error finding container 74a4e34f428da8a2ff556d8dfd1f97fee823168f0f90e2b9904219b1c391caa7: Status 404 returned error can't find the container with id 74a4e34f428da8a2ff556d8dfd1f97fee823168f0f90e2b9904219b1c391caa7 Nov 03 10:41:02.567110 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:02.567075 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" event={"ID":"ac3318e5-f88a-40e6-9482-55a0e47c162e","Type":"ContainerStarted","Data":"857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366"} Nov 03 10:41:02.567475 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:02.567130 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" event={"ID":"ac3318e5-f88a-40e6-9482-55a0e47c162e","Type":"ContainerStarted","Data":"74a4e34f428da8a2ff556d8dfd1f97fee823168f0f90e2b9904219b1c391caa7"} Nov 03 10:41:02.567475 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:02.567206 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:02.586654 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:02.586581 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" podStartSLOduration=1.156181478 podStartE2EDuration="1.58656003s" totalImagesPullingTime="430.378552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:02.584182072 +0000 UTC m=+740.569053355" watchObservedRunningTime="2026-11-03 10:41:02.58656003 +0000 UTC m=+740.571431313" Nov 03 10:41:08.578418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:08.578384 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 10:41:11.942788 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:11.942754 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm"] Nov 03 10:41:11.947221 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:11.947197 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:11.954869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:11.954837 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"default-dockercfg-lpsf5\"" Nov 03 10:41:11.955219 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:11.955186 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm"] Nov 03 10:41:11.955386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:11.955361 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"simulator-w3-b1c6f6-kserve-self-signed-certs\"" Nov 03 10:41:11.963617 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:11.963597 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"openshift-service-ca.crt\"" Nov 03 10:41:11.963730 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:11.963677 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"kube-root-ca.crt\"" Nov 03 10:41:12.049489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.049456 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-dshm\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.049489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.049493 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpx8q\" (UniqueName: \"kubernetes.io/projected/62e2a4eb-f888-4f51-8d56-a291557109d4-kube-api-access-mpx8q\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.049770 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.049515 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-home\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.049770 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.049669 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-model-cache\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.049770 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.049718 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e2a4eb-f888-4f51-8d56-a291557109d4-tls-certs\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.049925 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.049787 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-tmp-dir\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.150588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.150556 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-dshm\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.150756 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.150593 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpx8q\" (UniqueName: \"kubernetes.io/projected/62e2a4eb-f888-4f51-8d56-a291557109d4-kube-api-access-mpx8q\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.150756 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.150613 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-home\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.150756 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.150653 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-model-cache\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.150756 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.150674 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e2a4eb-f888-4f51-8d56-a291557109d4-tls-certs\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.150756 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.150694 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-tmp-dir\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.151091 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.151070 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-home\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.151166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.151115 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-model-cache\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.151166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.151140 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-tmp-dir\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.153017 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.152996 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-dshm\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.153240 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.153218 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e2a4eb-f888-4f51-8d56-a291557109d4-tls-certs\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.161100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.161072 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpx8q\" (UniqueName: \"kubernetes.io/projected/62e2a4eb-f888-4f51-8d56-a291557109d4-kube-api-access-mpx8q\") pod \"simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.263602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.263508 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:12.539901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.539871 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm"] Nov 03 10:41:12.541856 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:41:12.541827 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62e2a4eb_f888_4f51_8d56_a291557109d4.slice/crio-7ba3a48220cb08ef346d5a487ad04e87c962315afdd6b9dfb16f787c63975edd WatchSource:0}: Error finding container 7ba3a48220cb08ef346d5a487ad04e87c962315afdd6b9dfb16f787c63975edd: Status 404 returned error can't find the container with id 7ba3a48220cb08ef346d5a487ad04e87c962315afdd6b9dfb16f787c63975edd Nov 03 10:41:12.629182 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.629147 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" event={"ID":"62e2a4eb-f888-4f51-8d56-a291557109d4","Type":"ContainerStarted","Data":"9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996"} Nov 03 10:41:12.629381 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.629202 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" event={"ID":"62e2a4eb-f888-4f51-8d56-a291557109d4","Type":"ContainerStarted","Data":"7ba3a48220cb08ef346d5a487ad04e87c962315afdd6b9dfb16f787c63975edd"} Nov 03 10:41:12.645094 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:12.645032 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podStartSLOduration=1.645013171 podStartE2EDuration="1.645013171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:12.644223051 +0000 UTC m=+750.629094334" watchObservedRunningTime="2026-11-03 10:41:12.645013171 +0000 UTC m=+750.629884456" Nov 03 10:41:13.264521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:13.264487 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:21.528049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.528012 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf"] Nov 03 10:41:21.533371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.533347 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.540506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.540482 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"simulator-w0-f78621-kserve-self-signed-certs\"" Nov 03 10:41:21.540665 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.540513 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"kube-root-ca.crt\"" Nov 03 10:41:21.540844 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.540827 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"default-dockercfg-rp5ss\"" Nov 03 10:41:21.540905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.540877 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"openshift-service-ca.crt\"" Nov 03 10:41:21.542058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.542034 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf"] Nov 03 10:41:21.551462 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.551433 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-model-cache\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.551616 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.551465 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-tmp-dir\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.551616 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.551562 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-home\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.551616 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.551581 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-dshm\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.551805 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.551712 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf375e6-6ba5-4fca-b377-72060451aa71-tls-certs\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.551863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.551846 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99psf\" (UniqueName: \"kubernetes.io/projected/ddf375e6-6ba5-4fca-b377-72060451aa71-kube-api-access-99psf\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.653066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.653032 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-model-cache\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.653066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.653064 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-tmp-dir\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.653330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.653105 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-home\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.653330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.653121 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-dshm\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.653330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.653156 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf375e6-6ba5-4fca-b377-72060451aa71-tls-certs\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.653330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.653239 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99psf\" (UniqueName: \"kubernetes.io/projected/ddf375e6-6ba5-4fca-b377-72060451aa71-kube-api-access-99psf\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.653623 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.653555 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-model-cache\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.653623 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.653579 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-tmp-dir\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.653623 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.653608 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-home\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.655589 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.655571 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-dshm\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.655899 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.655879 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf375e6-6ba5-4fca-b377-72060451aa71-tls-certs\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.660163 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.660145 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99psf\" (UniqueName: \"kubernetes.io/projected/ddf375e6-6ba5-4fca-b377-72060451aa71-kube-api-access-99psf\") pod \"simulator-w0-f78621-kserve-7f98bbcc9-blwmf\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:21.849885 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:21.849852 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:22.126175 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:22.126141 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf"] Nov 03 10:41:22.127927 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:41:22.127890 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddf375e6_6ba5_4fca_b377_72060451aa71.slice/crio-320d7c0cdf7c11e7cab91e2e881db6cde4387058a748e920e3806f773cbcc3d2 WatchSource:0}: Error finding container 320d7c0cdf7c11e7cab91e2e881db6cde4387058a748e920e3806f773cbcc3d2: Status 404 returned error can't find the container with id 320d7c0cdf7c11e7cab91e2e881db6cde4387058a748e920e3806f773cbcc3d2 Nov 03 10:41:22.264554 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:22.264520 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:22.276993 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:22.276968 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:22.694345 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:22.694255 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" event={"ID":"ddf375e6-6ba5-4fca-b377-72060451aa71","Type":"ContainerStarted","Data":"91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204"} Nov 03 10:41:22.694345 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:22.694309 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" event={"ID":"ddf375e6-6ba5-4fca-b377-72060451aa71","Type":"ContainerStarted","Data":"320d7c0cdf7c11e7cab91e2e881db6cde4387058a748e920e3806f773cbcc3d2"} Nov 03 10:41:22.705711 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:22.705682 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:41:22.712018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:22.711965 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podStartSLOduration=1.7119502930000001 podStartE2EDuration="1.711950293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:22.711389266 +0000 UTC m=+760.696260550" watchObservedRunningTime="2026-11-03 10:41:22.711950293 +0000 UTC m=+760.696821574" Nov 03 10:41:22.850396 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:22.850362 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:24.273286 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:41:24.273242 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:41:31.850487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:31.850444 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:31.862974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:31.862948 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:32.771189 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:41:32.771160 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:41:54.273261 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:41:54.273222 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:42:24.275170 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:42:24.275130 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:42:31.817745 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.817700 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz"] Nov 03 10:42:31.821733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.821710 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.832977 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.832951 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b1c6f6-openshift-default-dockercfg-h8q5b\"" Nov 03 10:42:31.838254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.838223 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz"] Nov 03 10:42:31.870957 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.870914 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.871147 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.870966 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.871147 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.871107 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194ed661-f22d-48d6-ba45-016943dee4b0-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.871147 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.871142 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.871351 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.871206 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bpm\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-kube-api-access-f6bpm\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.871351 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.871237 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.871466 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.871414 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.871522 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.871484 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.871577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.871545 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194ed661-f22d-48d6-ba45-016943dee4b0-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.972464 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972428 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.972653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972472 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194ed661-f22d-48d6-ba45-016943dee4b0-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.972653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972498 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.972653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972514 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.972653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972561 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194ed661-f22d-48d6-ba45-016943dee4b0-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.972653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972581 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.972926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972679 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bpm\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-kube-api-access-f6bpm\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.972926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972718 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.972926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972809 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.973084 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.972935 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.973084 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.973016 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.973230 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.973208 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.973286 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.973268 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.973357 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.973287 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194ed661-f22d-48d6-ba45-016943dee4b0-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.974842 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.974823 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.975118 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.975099 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194ed661-f22d-48d6-ba45-016943dee4b0-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.980830 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.980802 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:31.980948 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:31.980903 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bpm\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-kube-api-access-f6bpm\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:32.146961 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:32.146878 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:32.439001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:32.438966 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz"] Nov 03 10:42:32.441326 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:42:32.441268 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194ed661_f22d_48d6_ba45_016943dee4b0.slice/crio-b4bb7c1d682ac6f89d050df63443ccc9d82f43b3d4faea591b8f5bf7052f759c WatchSource:0}: Error finding container b4bb7c1d682ac6f89d050df63443ccc9d82f43b3d4faea591b8f5bf7052f759c: Status 404 returned error can't find the container with id b4bb7c1d682ac6f89d050df63443ccc9d82f43b3d4faea591b8f5bf7052f759c Nov 03 10:42:32.444807 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:32.444759 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:42:32.444922 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:32.444861 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:42:32.444922 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:32.444908 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:42:33.147585 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:33.147544 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" event={"ID":"194ed661-f22d-48d6-ba45-016943dee4b0","Type":"ContainerStarted","Data":"196282c1b29cbc0d33d870975594f7ee7df6738e8218b85c0acda5fac6e76f30"} Nov 03 10:42:33.148038 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:33.147605 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" event={"ID":"194ed661-f22d-48d6-ba45-016943dee4b0","Type":"ContainerStarted","Data":"b4bb7c1d682ac6f89d050df63443ccc9d82f43b3d4faea591b8f5bf7052f759c"} Nov 03 10:42:33.171550 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:33.171487 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" podStartSLOduration=2.1714687870000002 podStartE2EDuration="2.171468787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:42:33.169106147 +0000 UTC m=+831.153977430" watchObservedRunningTime="2026-11-03 10:42:33.171468787 +0000 UTC m=+831.156340071" Nov 03 10:42:34.147087 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:34.147048 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:34.148737 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:34.148712 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 10:42:34.149114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:34.148759 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 10:42:35.148131 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:35.148088 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 10:42:35.148358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:35.148154 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 10:42:36.147637 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:36.147600 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 10:42:36.148049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:36.147656 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 10:42:37.147835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:37.147796 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 10:42:37.148471 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:37.147850 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 10:42:38.147850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:38.147815 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 10:42:38.148270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:38.147869 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 10:42:39.147734 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:39.147690 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 10:42:39.147924 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:39.147774 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 10:42:40.152349 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:40.152318 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:40.152755 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:40.152735 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:40.153363 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:40.153345 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:42:42.053534 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.053488 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t"] Nov 03 10:42:42.058705 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.058674 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.073992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.073962 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-010232-openshift-default-dockercfg-txh8q\"" Nov 03 10:42:42.088019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.087971 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t"] Nov 03 10:42:42.107735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.107698 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.107916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.107745 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh52z\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-kube-api-access-wh52z\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.107916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.107773 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.107916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.107895 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.108119 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.107951 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.108119 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.108084 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.108214 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.108126 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.108273 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.108242 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.108351 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.108289 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.209899 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.209870 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.210116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.209910 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.210116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.209939 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.210116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.210000 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.210116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.210028 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.210116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.210076 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.210116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.210116 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.210512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.210195 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.210580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.210518 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.211142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.210977 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.211142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.211035 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh52z\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-kube-api-access-wh52z\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.211142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.211036 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.211420 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.211162 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.211420 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.211261 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.212427 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.212402 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.213834 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.213807 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.217215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.217193 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.218577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.218548 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh52z\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-kube-api-access-wh52z\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.387823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.387785 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-010232-openshift-default-dockercfg-txh8q\"" Nov 03 10:42:42.395995 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.395962 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:42.700844 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.700817 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t"] Nov 03 10:42:42.703259 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:42:42.703232 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96bd9cc_e47f_40b3_ad46_2bd17d497592.slice/crio-11a58d907579b9dc3d134ffa521b06cf51659d4d49b6d55695c5fe0c76be6a1b WatchSource:0}: Error finding container 11a58d907579b9dc3d134ffa521b06cf51659d4d49b6d55695c5fe0c76be6a1b: Status 404 returned error can't find the container with id 11a58d907579b9dc3d134ffa521b06cf51659d4d49b6d55695c5fe0c76be6a1b Nov 03 10:42:42.706996 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.706957 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:42:42.707084 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.707037 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:42:42.707084 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:42.707069 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:42:43.237772 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.237707 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" event={"ID":"f96bd9cc-e47f-40b3-ad46-2bd17d497592","Type":"ContainerStarted","Data":"63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375"} Nov 03 10:42:43.238254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.237778 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" event={"ID":"f96bd9cc-e47f-40b3-ad46-2bd17d497592","Type":"ContainerStarted","Data":"11a58d907579b9dc3d134ffa521b06cf51659d4d49b6d55695c5fe0c76be6a1b"} Nov 03 10:42:43.262024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.261958 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" podStartSLOduration=1.261939314 podStartE2EDuration="1.261939314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:42:43.258693013 +0000 UTC m=+841.243564318" watchObservedRunningTime="2026-11-03 10:42:43.261939314 +0000 UTC m=+841.246810599" Nov 03 10:42:43.397079 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.397040 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:43.398384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.398310 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 10:42:43.398384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.398372 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 10:42:43.742953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.742920 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p"] Nov 03 10:42:43.747421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.747396 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.758160 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.758119 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 10:42:43.758644 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.758147 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 10:42:43.758644 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.758201 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 10:42:43.758644 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.758230 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-49a4ab2b36b3\"" Nov 03 10:42:43.758644 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.758573 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 10:42:43.758644 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.758620 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-83bc832de857\"" Nov 03 10:42:43.759079 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.759059 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-77sqt\"" Nov 03 10:42:43.762333 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.762308 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p"] Nov 03 10:42:43.846038 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.846001 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-plugins-config-volume\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.846038 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.846049 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-provider-credentials\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.846472 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.846442 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-routing-overlay-volume\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.846553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.846490 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4wzb\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-kube-api-access-c4wzb\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.950662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.950616 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-plugins-config-volume\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.950937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.950915 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-provider-credentials\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.951248 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.951227 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-routing-overlay-volume\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.951405 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.951387 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4wzb\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-kube-api-access-c4wzb\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.956367 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.952811 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-plugins-config-volume\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.957714 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.957689 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-routing-overlay-volume\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.960021 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.959999 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-provider-credentials\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:43.966207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:43.966186 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4wzb\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-kube-api-access-c4wzb\") pod \"payload-processing-external-model-5bc8c6dfc6-w8x5p\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:44.063008 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:44.062918 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:44.396471 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:44.396426 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 10:42:44.396993 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:44.396495 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 10:42:44.475176 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:44.474345 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p"] Nov 03 10:42:44.990025 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:44.989986 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt"] Nov 03 10:42:44.995471 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:44.995449 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.006987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.004841 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e78c33-openshift-default-dockercfg-brqpb\"" Nov 03 10:42:45.010626 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.010393 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt"] Nov 03 10:42:45.098738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.096678 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.098738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.097413 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.098738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.097518 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.098738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.097552 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.098738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.097602 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btlbj\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-kube-api-access-btlbj\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.098738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.097669 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3adb96-ea20-407b-bad4-7401e60bfd27-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.098738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.097696 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.098738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.097744 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.098738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.097796 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.209355 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.213084 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.213154 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.213224 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btlbj\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-kube-api-access-btlbj\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.213336 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3adb96-ea20-407b-bad4-7401e60bfd27-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.213370 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.213454 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.213519 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.213627 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.211800 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.215463 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.215764 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.215895 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.229318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.216128 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3adb96-ea20-407b-bad4-7401e60bfd27-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.245004 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.231064 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.267900 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.267833 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.270991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.270959 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.297247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.296054 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btlbj\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-kube-api-access-btlbj\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.395334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.378552 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:45.395334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.383270 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" event={"ID":"46123560-a0e7-4041-bd5e-6f8efd230053","Type":"ContainerStarted","Data":"508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4"} Nov 03 10:42:45.395334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.383350 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" event={"ID":"46123560-a0e7-4041-bd5e-6f8efd230053","Type":"ContainerStarted","Data":"1abaab2ebfd73d243c20fa13c3d2c1a27a355e87ffba46feb1705107e52113d0"} Nov 03 10:42:45.395334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.383411 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:42:45.400199 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.398863 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 10:42:45.400199 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.398974 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 10:42:45.419929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:45.417119 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" podStartSLOduration=2.41709772 podStartE2EDuration="2.41709772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:42:45.415595811 +0000 UTC m=+843.400467094" watchObservedRunningTime="2026-11-03 10:42:45.41709772 +0000 UTC m=+843.401969004" Nov 03 10:42:46.435524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:46.420968 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 10:42:46.435524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:46.421025 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 10:42:46.898724 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:42:46.898676 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3adb96_ea20_407b_bad4_7401e60bfd27.slice/crio-3679d61e30d5641819af2941e62fd24b90e1bf5d4fe3317f50e51122ead32908 WatchSource:0}: Error finding container 3679d61e30d5641819af2941e62fd24b90e1bf5d4fe3317f50e51122ead32908: Status 404 returned error can't find the container with id 3679d61e30d5641819af2941e62fd24b90e1bf5d4fe3317f50e51122ead32908 Nov 03 10:42:46.998119 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:46.998034 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt"] Nov 03 10:42:47.001511 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:47.001467 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:42:47.001736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:47.001696 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:42:47.002591 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:47.002563 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:42:47.397597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:47.397559 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 10:42:47.397793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:47.397754 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 10:42:47.655578 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:47.655534 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" event={"ID":"ee3adb96-ea20-407b-bad4-7401e60bfd27","Type":"ContainerStarted","Data":"113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517"} Nov 03 10:42:47.656081 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:47.655605 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" event={"ID":"ee3adb96-ea20-407b-bad4-7401e60bfd27","Type":"ContainerStarted","Data":"3679d61e30d5641819af2941e62fd24b90e1bf5d4fe3317f50e51122ead32908"} Nov 03 10:42:47.677614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:47.677558 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" podStartSLOduration=3.677542079 podStartE2EDuration="3.677542079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:42:47.675790135 +0000 UTC m=+845.660661419" watchObservedRunningTime="2026-11-03 10:42:47.677542079 +0000 UTC m=+845.662413361" Nov 03 10:42:48.383832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:48.383802 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:48.385589 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:48.385564 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 10:42:48.385719 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:48.385618 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 10:42:48.396534 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:48.396502 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 10:42:48.396676 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:48.396558 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 10:42:49.384685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:49.384645 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 10:42:49.385110 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:49.384700 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 10:42:49.396493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:49.396456 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 10:42:49.396652 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:49.396509 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 10:42:50.384052 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:50.384009 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 10:42:50.384236 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:50.384078 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 10:42:50.401453 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:50.401416 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:50.401943 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:50.401816 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:50.402499 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:50.402475 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 10:42:51.384122 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:51.384077 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 10:42:51.384328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:51.384149 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 10:42:52.388319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:52.387726 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 10:42:52.388319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:52.387794 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 10:42:53.384789 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:53.384747 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 10:42:53.385075 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:53.385046 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 10:42:54.274581 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:42:54.274539 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:42:54.388504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:54.388474 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:54.388905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:54.388880 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:54.389544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:54.389526 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 10:42:57.676568 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:42:57.676523 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:43:12.448326 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.448274 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq"] Nov 03 10:43:12.452664 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.452634 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.461493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.461454 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d5d3c3-openshift-default-dockercfg-wbpds\"" Nov 03 10:43:12.465156 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.465116 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq"] Nov 03 10:43:12.597843 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.597802 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.597843 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.597843 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4973c07-e22e-4d8a-be06-505b9d3b7297-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.598070 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.597876 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.598070 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.597953 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.598070 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.597980 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.598070 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.598008 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.598209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.598071 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.598209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.598112 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gpbm\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-kube-api-access-9gpbm\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.598209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.598177 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.699821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.699741 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.699821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.699796 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gpbm\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-kube-api-access-9gpbm\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.699838 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.699909 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.699925 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4973c07-e22e-4d8a-be06-505b9d3b7297-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.699955 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.699996 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.700032 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.700072 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.700254 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.700428 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.700901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.700557 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.701159 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.701015 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.701716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.701691 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4973c07-e22e-4d8a-be06-505b9d3b7297-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.702781 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.702761 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.703225 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.703205 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.708692 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.708665 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gpbm\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-kube-api-access-9gpbm\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.708781 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.708669 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:12.769836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:12.769520 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:13.057172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:13.057081 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq"] Nov 03 10:43:13.060097 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:43:13.060066 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4973c07_e22e_4d8a_be06_505b9d3b7297.slice/crio-a4be66d17f0ac18258177401ed0f200ef68961ba7c7fa0ba9f0afdc747db7ea1 WatchSource:0}: Error finding container a4be66d17f0ac18258177401ed0f200ef68961ba7c7fa0ba9f0afdc747db7ea1: Status 404 returned error can't find the container with id a4be66d17f0ac18258177401ed0f200ef68961ba7c7fa0ba9f0afdc747db7ea1 Nov 03 10:43:13.062680 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:13.062645 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:13.062763 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:13.062725 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:13.062763 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:13.062755 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:13.286480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:13.286441 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p"] Nov 03 10:43:13.939394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:13.939352 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" event={"ID":"c4973c07-e22e-4d8a-be06-505b9d3b7297","Type":"ContainerStarted","Data":"4d745b22ef11aa17901fde13c8616b1e7a9af77f82ccac53950594fe9603e368"} Nov 03 10:43:13.940050 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:13.940023 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" event={"ID":"c4973c07-e22e-4d8a-be06-505b9d3b7297","Type":"ContainerStarted","Data":"a4be66d17f0ac18258177401ed0f200ef68961ba7c7fa0ba9f0afdc747db7ea1"} Nov 03 10:43:13.962840 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:13.962761 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" podStartSLOduration=1.962741802 podStartE2EDuration="1.962741802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:13.958660707 +0000 UTC m=+871.943531991" watchObservedRunningTime="2026-11-03 10:43:13.962741802 +0000 UTC m=+871.947613084" Nov 03 10:43:14.769715 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:14.769678 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:14.771717 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:14.771682 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 10:43:14.771855 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:14.771748 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 10:43:15.771193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:15.771158 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 10:43:15.771643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:15.771209 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 10:43:16.770066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.770028 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 10:43:16.770251 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.770084 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 10:43:16.885832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.885792 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb"] Nov 03 10:43:16.891322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.891285 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:16.909699 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.909660 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb"] Nov 03 10:43:16.972712 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.972671 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:16.972940 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.972727 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hjcw\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-kube-api-access-2hjcw\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:16.972940 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.972916 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:16.973068 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.973050 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:16.973122 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.973080 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:16.973188 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.973124 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:16.973188 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.973153 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:16.973308 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.973243 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:16.973308 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:16.973279 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.074743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.074713 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.074929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.074747 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.074929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.074798 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.074929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.074821 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hjcw\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-kube-api-access-2hjcw\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.074929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.074924 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.075180 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.075015 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.075180 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.075047 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.075180 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.075071 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.075180 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.075099 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.075447 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.075222 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.075536 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.075505 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.075601 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.075533 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.075657 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.075598 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.075724 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.075662 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.077373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.077352 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.077580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.077563 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.083786 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.083761 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.083903 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.083889 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hjcw\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-kube-api-access-2hjcw\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.210522 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.210486 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:17.542332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.538575 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb"] Nov 03 10:43:17.544779 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.544173 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:17.544779 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.544264 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:17.544779 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.544343 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:17.770606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.770189 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 10:43:17.770606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.770246 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 10:43:17.975837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.975805 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" event={"ID":"3146c2f2-37ed-4205-b4e1-ccb52a24f433","Type":"ContainerStarted","Data":"e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65"} Nov 03 10:43:17.976493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.975866 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" event={"ID":"3146c2f2-37ed-4205-b4e1-ccb52a24f433","Type":"ContainerStarted","Data":"60bef3ff74d931e1f1fc88face661065733829400d69a497745ae8e320444cda"} Nov 03 10:43:18.000027 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:17.999950 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" podStartSLOduration=1.999931004 podStartE2EDuration="1.999931004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:17.994032271 +0000 UTC m=+875.978903554" watchObservedRunningTime="2026-11-03 10:43:17.999931004 +0000 UTC m=+875.984802288" Nov 03 10:43:18.210793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.210756 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:18.212234 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.212207 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 10:43:18.212390 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.212265 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 10:43:18.287342 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.287287 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" podUID="46123560-a0e7-4041-bd5e-6f8efd230053" containerName="payload-processing" containerID="cri-o://508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4" gracePeriod=25 Nov 03 10:43:18.704622 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.704589 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:43:18.776650 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.776606 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 10:43:18.776844 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.776673 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 10:43:18.800573 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.800486 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-routing-overlay-volume\") pod \"46123560-a0e7-4041-bd5e-6f8efd230053\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " Nov 03 10:43:18.800769 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.800756 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-kube-api-access-c4wzb\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-kube-api-access-c4wzb\") pod \"46123560-a0e7-4041-bd5e-6f8efd230053\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " Nov 03 10:43:18.800848 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.800823 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-routing-overlay-volume" pod "46123560-a0e7-4041-bd5e-6f8efd230053" (UID: "46123560-a0e7-4041-bd5e-6f8efd230053"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:43:18.800977 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.800965 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-provider-credentials\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-provider-credentials\") pod \"46123560-a0e7-4041-bd5e-6f8efd230053\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " Nov 03 10:43:18.801063 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.801054 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-plugins-config-volume\") pod \"46123560-a0e7-4041-bd5e-6f8efd230053\" (UID: \"46123560-a0e7-4041-bd5e-6f8efd230053\") " Nov 03 10:43:18.801557 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.801536 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-plugins-config-volume" pod "46123560-a0e7-4041-bd5e-6f8efd230053" (UID: "46123560-a0e7-4041-bd5e-6f8efd230053"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:43:18.801724 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.801710 5086 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:43:18.803104 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.803086 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-provider-credentials" pod "46123560-a0e7-4041-bd5e-6f8efd230053" (UID: "46123560-a0e7-4041-bd5e-6f8efd230053"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:43:18.803227 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.803208 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-kube-api-access-c4wzb" pod "46123560-a0e7-4041-bd5e-6f8efd230053" (UID: "46123560-a0e7-4041-bd5e-6f8efd230053"). InnerVolumeSpecName "kube-api-access-c4wzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:43:18.902920 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.902887 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4wzb\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-kube-api-access-c4wzb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:43:18.902920 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.902919 5086 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46123560-a0e7-4041-bd5e-6f8efd230053-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:43:18.903129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.902933 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46123560-a0e7-4041-bd5e-6f8efd230053-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:43:18.965099 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.965046 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw"] Nov 03 10:43:18.966005 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.965979 5086 cpu_manager.go:414] "containerMap: removing container" podUID="46123560-a0e7-4041-bd5e-6f8efd230053" containerName="payload-processing" Nov 03 10:43:18.966005 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.966004 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46123560-a0e7-4041-bd5e-6f8efd230053" containerName="payload-processing" Nov 03 10:43:18.966173 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.966159 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="46123560-a0e7-4041-bd5e-6f8efd230053" containerName="payload-processing" Nov 03 10:43:18.971239 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.970492 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:18.974337 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.973948 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw"] Nov 03 10:43:18.974337 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.974255 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 10:43:18.974936 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.974579 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 10:43:18.974936 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.974627 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fbzlj\"" Nov 03 10:43:18.974936 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.974666 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 10:43:18.991030 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.990990 5086 generic.go:356] "Generic (PLEG): container finished" podID="46123560-a0e7-4041-bd5e-6f8efd230053" containerID="508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4" exitCode=0 Nov 03 10:43:18.991489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.991249 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" Nov 03 10:43:18.991723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.991699 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" event={"ID":"46123560-a0e7-4041-bd5e-6f8efd230053","Type":"ContainerDied","Data":"508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4"} Nov 03 10:43:18.991819 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.991757 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p" event={"ID":"46123560-a0e7-4041-bd5e-6f8efd230053","Type":"ContainerDied","Data":"1abaab2ebfd73d243c20fa13c3d2c1a27a355e87ffba46feb1705107e52113d0"} Nov 03 10:43:18.991819 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:18.991790 5086 scope.go:122] "RemoveContainer" containerID="508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4" Nov 03 10:43:19.004262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.003072 5086 scope.go:122] "RemoveContainer" containerID="508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4" Nov 03 10:43:19.004262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.003714 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4ec4160-f315-4999-bdb3-a24364695cff-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.004262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.003800 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4ec4160-f315-4999-bdb3-a24364695cff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.004262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.003919 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ec4160-f315-4999-bdb3-a24364695cff-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.004262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.003947 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4ec4160-f315-4999-bdb3-a24364695cff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.004262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.003979 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4sb9\" (UniqueName: \"kubernetes.io/projected/b4ec4160-f315-4999-bdb3-a24364695cff-kube-api-access-x4sb9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.004262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.004022 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4ec4160-f315-4999-bdb3-a24364695cff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.004262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.004118 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ec4160-f315-4999-bdb3-a24364695cff-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.005459 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:43:19.005425 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4\": container with ID starting with 508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4 not found: ID does not exist" containerID="508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4" Nov 03 10:43:19.005552 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.005462 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4"} err="failed to get container status \"508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4\": rpc error: code = NotFound desc = could not find container \"508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4\": container with ID starting with 508e77778235a0340b2372f51bcd128f4ac8df7df64303702022b46aace715c4 not found: ID does not exist" Nov 03 10:43:19.050780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.050696 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p"] Nov 03 10:43:19.053570 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.053532 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc8c6dfc6-w8x5p"] Nov 03 10:43:19.105610 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.105578 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4ec4160-f315-4999-bdb3-a24364695cff-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.105809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.105653 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4ec4160-f315-4999-bdb3-a24364695cff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.105809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.105699 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ec4160-f315-4999-bdb3-a24364695cff-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.105809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.105725 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4ec4160-f315-4999-bdb3-a24364695cff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.105809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.105753 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4sb9\" (UniqueName: \"kubernetes.io/projected/b4ec4160-f315-4999-bdb3-a24364695cff-kube-api-access-x4sb9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.106025 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.105845 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4ec4160-f315-4999-bdb3-a24364695cff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.106087 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.106014 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ec4160-f315-4999-bdb3-a24364695cff-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.107328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.107271 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4ec4160-f315-4999-bdb3-a24364695cff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.108803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.108778 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4ec4160-f315-4999-bdb3-a24364695cff-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.109088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.109059 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ec4160-f315-4999-bdb3-a24364695cff-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.109189 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.109112 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4ec4160-f315-4999-bdb3-a24364695cff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.109274 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.109253 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4ec4160-f315-4999-bdb3-a24364695cff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.113419 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.113395 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ec4160-f315-4999-bdb3-a24364695cff-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.113778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.113758 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4sb9\" (UniqueName: \"kubernetes.io/projected/b4ec4160-f315-4999-bdb3-a24364695cff-kube-api-access-x4sb9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mtdw\" (UID: \"b4ec4160-f315-4999-bdb3-a24364695cff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.211155 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.211111 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 10:43:19.211426 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.211171 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 10:43:19.288974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.288938 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:19.696401 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:43:19.696364 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ec4160_f315_4999_bdb3_a24364695cff.slice/crio-1635a75aff816534d0b56efccba6c266dae57e29372894cceda09e46fed14554 WatchSource:0}: Error finding container 1635a75aff816534d0b56efccba6c266dae57e29372894cceda09e46fed14554: Status 404 returned error can't find the container with id 1635a75aff816534d0b56efccba6c266dae57e29372894cceda09e46fed14554 Nov 03 10:43:19.698641 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.698583 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw"] Nov 03 10:43:19.989110 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.989024 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:19.989361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.989340 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:19.990026 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.990006 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 10:43:19.998746 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:19.998722 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" event={"ID":"b4ec4160-f315-4999-bdb3-a24364695cff","Type":"ContainerStarted","Data":"1635a75aff816534d0b56efccba6c266dae57e29372894cceda09e46fed14554"} Nov 03 10:43:20.173774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:20.173741 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46123560-a0e7-4041-bd5e-6f8efd230053" path="/var/lib/kubelet/pods/46123560-a0e7-4041-bd5e-6f8efd230053/volumes" Nov 03 10:43:20.211758 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:20.211725 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 10:43:20.211937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:20.211773 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 10:43:21.211155 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:21.211123 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 10:43:21.211550 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:21.211184 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 10:43:22.211783 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:22.211752 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 10:43:22.212256 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:22.212228 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 10:43:23.211489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:23.211454 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 10:43:23.211683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:23.211530 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 10:43:24.211764 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:24.211726 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 10:43:24.212245 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:24.211790 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 10:43:24.279963 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:43:24.279921 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:43:24.561820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:24.561151 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:24.561820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:24.561370 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:25.071562 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:25.071524 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" event={"ID":"b4ec4160-f315-4999-bdb3-a24364695cff","Type":"ContainerStarted","Data":"680cbbcd0f9027d85cc9d42823b606a8b22c4328c68a9f3f61f99c8923e6c420"} Nov 03 10:43:25.071803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:25.071785 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:25.073953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:25.073926 5086 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9mtdw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 10:43:25.074156 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:25.074131 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" podUID="b4ec4160-f315-4999-bdb3-a24364695cff" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 10:43:25.289320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:25.288974 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:25.294070 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:25.291829 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:25.294070 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:25.292731 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 10:43:25.314753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:25.314676 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" podStartSLOduration=2.453969654 podStartE2EDuration="7.314660273s" totalImagesPullingTime="4.860690619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:25.091181236 +0000 UTC m=+883.076052520" watchObservedRunningTime="2026-11-03 10:43:25.314660273 +0000 UTC m=+883.299531556" Nov 03 10:43:26.077277 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:26.077248 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mtdw" Nov 03 10:43:42.263599 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:42.263556 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:43:42.267204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:42.267178 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:43:42.270090 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:42.270063 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:43:42.275422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:42.275397 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:43:45.080420 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.080385 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg"] Nov 03 10:43:45.091034 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.091001 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg"] Nov 03 10:43:45.091239 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.091127 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.120237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.120205 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.120458 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.120242 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgmp\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-kube-api-access-sqgmp\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.120458 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.120375 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.120458 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.120436 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.120632 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.120495 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.120632 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.120562 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.120632 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.120596 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.120797 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.120721 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.120797 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.120768 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.222355 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.222315 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.222561 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.222374 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgmp\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-kube-api-access-sqgmp\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.222561 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.222458 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.222700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.222567 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.222700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.222591 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.222700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.222621 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.222700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.222663 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.222909 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.222730 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.222909 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.222791 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.223055 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.223025 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.223195 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.223158 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.223260 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.223213 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.223554 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.223528 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.224475 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.224448 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.225288 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.225265 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.227206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.227186 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.230202 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.230184 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.230513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.230479 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgmp\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-kube-api-access-sqgmp\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.436877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.436788 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:45.721888 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.721861 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg"] Nov 03 10:43:45.723862 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:43:45.723833 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f5f3e8_9726_44ef_9bff_68e68c8e35b8.slice/crio-35cf508936b9e27a9f5239f40c0267ec5f4835f5451b54b208f071f34092c5cd WatchSource:0}: Error finding container 35cf508936b9e27a9f5239f40c0267ec5f4835f5451b54b208f071f34092c5cd: Status 404 returned error can't find the container with id 35cf508936b9e27a9f5239f40c0267ec5f4835f5451b54b208f071f34092c5cd Nov 03 10:43:45.726614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.726578 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:45.726700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.726652 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:45.726700 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:45.726683 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:43:46.299971 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:46.299935 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" event={"ID":"74f5f3e8-9726-44ef-9bff-68e68c8e35b8","Type":"ContainerStarted","Data":"a6273df66ad5053a9eb0b17d8f5da960e62743223aa41e2eed6d9fde671a55e5"} Nov 03 10:43:46.300581 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:46.300550 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" event={"ID":"74f5f3e8-9726-44ef-9bff-68e68c8e35b8","Type":"ContainerStarted","Data":"35cf508936b9e27a9f5239f40c0267ec5f4835f5451b54b208f071f34092c5cd"} Nov 03 10:43:46.321530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:46.321450 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" podStartSLOduration=1.321425821 podStartE2EDuration="1.321425821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:46.318893406 +0000 UTC m=+904.303764691" watchObservedRunningTime="2026-11-03 10:43:46.321425821 +0000 UTC m=+904.306297132" Nov 03 10:43:46.441152 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:46.439510 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:46.444774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:46.444745 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 10:43:46.444915 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:46.444808 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 10:43:47.438318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:47.437708 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 10:43:47.438318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:47.437774 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 10:43:48.439832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:48.439794 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 10:43:48.449515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:48.449475 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 10:43:49.447759 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:49.447721 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 10:43:49.448842 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:49.448793 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 10:43:50.440735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:50.440694 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 10:43:50.440946 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:50.440771 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 10:43:51.437737 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:51.437699 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 10:43:51.438405 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:51.438371 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 10:43:52.437832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:52.437798 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 10:43:52.438238 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:52.437852 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 10:43:54.007076 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:54.007043 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:54.007582 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:54.007355 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:54.008252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:43:54.008231 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 10:43:54.286061 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:43:54.285941 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:44:24.488379 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:44:24.488063 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:44:30.114618 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.114094 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw"] Nov 03 10:44:30.120591 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.120563 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.137616 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.137587 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw"] Nov 03 10:44:30.179567 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.179527 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.179735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.179612 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.179735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.179690 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.179735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.179714 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.179735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.179736 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.179904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.179832 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.179904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.179869 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdz8\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-kube-api-access-tpdz8\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.179904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.179892 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.180034 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.179992 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.280856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.280822 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.280856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.280863 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281104 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.280896 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281104 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281008 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdz8\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-kube-api-access-tpdz8\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281104 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281054 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281104 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281101 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281408 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281213 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281408 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281249 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281408 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281358 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281576 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281425 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281659 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281978 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281944 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.281978 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.281964 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.282168 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.282139 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.283400 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.283367 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.284094 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.284070 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.289056 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.289027 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.289181 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.289165 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdz8\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-kube-api-access-tpdz8\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.450612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.450525 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:30.738139 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.738110 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw"] Nov 03 10:44:30.739109 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:44:30.739086 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf053e8e0_87a9_49fc_ae65_5c07b771b89e.slice/crio-07eaeeeecc74925a903488912eb37c29a6f09bf68fca3cdc408ccf899b7770bc WatchSource:0}: Error finding container 07eaeeeecc74925a903488912eb37c29a6f09bf68fca3cdc408ccf899b7770bc: Status 404 returned error can't find the container with id 07eaeeeecc74925a903488912eb37c29a6f09bf68fca3cdc408ccf899b7770bc Nov 03 10:44:30.741934 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.741898 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:30.742021 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.741970 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:30.742021 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:30.742007 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:31.746195 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:31.746154 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" event={"ID":"f053e8e0-87a9-49fc-ae65-5c07b771b89e","Type":"ContainerStarted","Data":"9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a"} Nov 03 10:44:31.746708 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:31.746208 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" event={"ID":"f053e8e0-87a9-49fc-ae65-5c07b771b89e","Type":"ContainerStarted","Data":"07eaeeeecc74925a903488912eb37c29a6f09bf68fca3cdc408ccf899b7770bc"} Nov 03 10:44:31.768307 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:31.768253 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" podStartSLOduration=1.768238757 podStartE2EDuration="1.768238757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:31.764968918 +0000 UTC m=+949.749840201" watchObservedRunningTime="2026-11-03 10:44:31.768238757 +0000 UTC m=+949.753110040" Nov 03 10:44:32.014751 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.014676 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g"] Nov 03 10:44:32.049318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.037654 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.065279 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.061672 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g"] Nov 03 10:44:32.105469 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.105432 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.105663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.105484 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.105663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.105523 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.105663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.105588 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.105663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.105654 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfdqt\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-kube-api-access-tfdqt\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.105903 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.105690 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.105903 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.105707 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.105903 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.105735 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.105903 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.105756 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207359 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207202 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207359 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207284 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfdqt\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-kube-api-access-tfdqt\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207369 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207396 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207432 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207460 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207504 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207542 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207590 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.207979 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.207902 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.208233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.208209 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.208423 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.208403 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.208622 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.208599 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.208847 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.208826 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.210253 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.210224 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.210441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.210422 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.225918 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.225886 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.247169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.247088 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfdqt\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-kube-api-access-tfdqt\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.370472 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.370431 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:32.451381 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.451350 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:32.455137 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.455105 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 10:44:32.455248 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.455172 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 10:44:32.686226 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.686197 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g"] Nov 03 10:44:32.688961 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:44:32.688938 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1980046_fc71_4a79_aef8_bc8b5a6dd3ee.slice/crio-d82e317585b2d8f879fcfd3a3e2943b298f62cc9164732cb8bdf5293325885b9 WatchSource:0}: Error finding container d82e317585b2d8f879fcfd3a3e2943b298f62cc9164732cb8bdf5293325885b9: Status 404 returned error can't find the container with id d82e317585b2d8f879fcfd3a3e2943b298f62cc9164732cb8bdf5293325885b9 Nov 03 10:44:32.692233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.692200 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:32.692372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.692263 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:32.692372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.692311 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:32.753372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:32.753335 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" event={"ID":"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee","Type":"ContainerStarted","Data":"d82e317585b2d8f879fcfd3a3e2943b298f62cc9164732cb8bdf5293325885b9"} Nov 03 10:44:33.458320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:33.457712 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 10:44:33.458320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:33.457775 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 10:44:33.772221 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:33.772124 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" event={"ID":"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee","Type":"ContainerStarted","Data":"3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6"} Nov 03 10:44:33.793125 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:33.793076 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" podStartSLOduration=2.793062231 podStartE2EDuration="2.793062231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:33.791344753 +0000 UTC m=+951.776216050" watchObservedRunningTime="2026-11-03 10:44:33.793062231 +0000 UTC m=+951.777933511" Nov 03 10:44:34.371369 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:34.371335 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:34.372927 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:34.372905 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 10:44:34.373014 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:34.372952 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 10:44:34.453319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:34.452735 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 10:44:34.453319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:34.452801 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 10:44:35.371850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:35.371757 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 10:44:35.371850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:35.371828 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 10:44:35.453772 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:35.453731 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 10:44:35.454057 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:35.454028 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 10:44:36.371870 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:36.371825 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 10:44:36.372418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:36.371904 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 10:44:36.475450 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:36.475411 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 10:44:36.476197 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:36.475778 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 10:44:37.371042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:37.371005 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 10:44:37.371247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:37.371063 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 10:44:37.451713 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:37.451671 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 10:44:37.452120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:37.451727 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 10:44:38.371878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:38.371844 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 10:44:38.372064 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:38.371899 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 10:44:38.457424 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:38.457393 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:38.457885 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:38.457759 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:38.458476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:38.458454 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 10:44:39.371067 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:39.371029 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 10:44:39.371269 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:39.371085 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 10:44:40.375972 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:40.375942 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:40.376403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:40.376259 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:40.376953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:40.376936 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 10:44:42.682977 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.682938 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4"] Nov 03 10:44:42.695522 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.695484 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4"] Nov 03 10:44:42.695841 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.695820 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.709548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.708505 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 10:44:42.846671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.831388 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.846671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.831504 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.846671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.831544 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.846671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.831584 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.846671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.831654 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.846671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.831702 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vmw\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-kube-api-access-54vmw\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.846671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.831777 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.846671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.831828 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.846671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.831907 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.937600 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.868831 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 10:44:42.985553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.985502 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.985874 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.985848 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.986101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.986077 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.986278 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.986259 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.986583 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.986561 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.986723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.986707 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.986850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.986834 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.987029 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.987010 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:42.987181 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:42.987165 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vmw\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-kube-api-access-54vmw\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.012321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.003063 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.012321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.003424 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.012321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.003676 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.013028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.012998 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.017005 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.016976 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.108348 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.061079 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 10:44:43.265369 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.265042 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.286321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.274061 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 10:44:43.361334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.319881 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.361334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.342661 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vmw\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-kube-api-access-54vmw\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.443844 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.443756 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:43.650320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:43.624231 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:44.063594 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:44.063539 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 10:44:44.090040 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:44.090007 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:44:45.111511 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:45.111408 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 10:44:45.373360 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:45.372825 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:44:45.376640 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:45.376584 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:45.376949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:45.376878 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:45.377143 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:45.377074 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:45.438726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:45.438692 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4"] Nov 03 10:44:45.613950 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:45.613867 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" event={"ID":"42d1ce75-5d3e-440f-8479-5ab13fa736e5","Type":"ContainerStarted","Data":"12a75edeed993c85d3843b05da79ea11c33ba1e89b34dfc373da1485dd351842"} Nov 03 10:44:46.635321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:46.626530 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" event={"ID":"42d1ce75-5d3e-440f-8479-5ab13fa736e5","Type":"ContainerStarted","Data":"3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070"} Nov 03 10:44:46.647170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:46.646938 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:46.648473 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:46.648444 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 10:44:46.648705 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:46.648672 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 10:44:46.649823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:46.649781 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" podStartSLOduration=4.64976762 podStartE2EDuration="4.64976762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:46.646837716 +0000 UTC m=+964.631708999" watchObservedRunningTime="2026-11-03 10:44:46.64976762 +0000 UTC m=+964.634638903" Nov 03 10:44:47.641322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:47.640699 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 10:44:47.641322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:47.640763 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 10:44:48.640712 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:48.640658 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 10:44:48.641037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:48.640970 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 10:44:49.641794 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:49.641758 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 10:44:49.642343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:49.641824 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 10:44:50.636839 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:50.636805 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 10:44:50.637103 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:50.637070 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 10:44:51.640319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:51.638390 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 10:44:51.640319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:51.638477 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 10:44:52.637475 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:52.637440 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 10:44:52.637646 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:52.637489 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 10:44:53.641402 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:53.641362 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:53.641881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:53.641772 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:53.642596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:53.642578 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 10:44:54.275833 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:44:54.275793 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:44:57.640801 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.640767 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9"] Nov 03 10:44:57.647802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.647775 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.669746 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.669712 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9"] Nov 03 10:44:57.707445 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.707396 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45lm\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-kube-api-access-g45lm\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.707636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.707563 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.707636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.707603 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.707731 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.707632 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.707789 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.707767 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.707836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.707804 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02267306-b3b0-43d5-a639-e04d7dd016b6-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.707918 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.707895 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.708076 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.708049 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.708137 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.708122 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.809728 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.809695 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.809904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.809747 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.809904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.809797 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45lm\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-kube-api-access-g45lm\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.809999 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.809914 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.809999 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.809959 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.810100 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.810065 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.810154 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.810130 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.810154 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.810147 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02267306-b3b0-43d5-a639-e04d7dd016b6-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.810262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.810174 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.810394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.810369 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.810599 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.810417 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.810775 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.810603 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.810858 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.810634 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.810913 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.810885 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02267306-b3b0-43d5-a639-e04d7dd016b6-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.812797 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.812769 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.813112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.813086 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.821983 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.821950 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.823414 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.823393 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45lm\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-kube-api-access-g45lm\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:57.974146 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:57.974057 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:58.257535 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:58.257510 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9"] Nov 03 10:44:58.259052 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:44:58.259023 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02267306_b3b0_43d5_a639_e04d7dd016b6.slice/crio-8d9ab4b6322a1d55646d34c6bbb18dfa0763536cc4a9fe5c65792a1eb808d96e WatchSource:0}: Error finding container 8d9ab4b6322a1d55646d34c6bbb18dfa0763536cc4a9fe5c65792a1eb808d96e: Status 404 returned error can't find the container with id 8d9ab4b6322a1d55646d34c6bbb18dfa0763536cc4a9fe5c65792a1eb808d96e Nov 03 10:44:58.261764 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:58.261722 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:58.261857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:58.261809 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:58.261897 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:58.261851 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:44:58.731674 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:58.731639 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" event={"ID":"02267306-b3b0-43d5-a639-e04d7dd016b6","Type":"ContainerStarted","Data":"5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9"} Nov 03 10:44:58.732092 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:58.731705 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" event={"ID":"02267306-b3b0-43d5-a639-e04d7dd016b6","Type":"ContainerStarted","Data":"8d9ab4b6322a1d55646d34c6bbb18dfa0763536cc4a9fe5c65792a1eb808d96e"} Nov 03 10:44:58.753882 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:58.753823 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podStartSLOduration=1.753808184 podStartE2EDuration="1.753808184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:58.750872967 +0000 UTC m=+976.735744251" watchObservedRunningTime="2026-11-03 10:44:58.753808184 +0000 UTC m=+976.738679470" Nov 03 10:44:58.974844 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:58.974808 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:44:59.991394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:59.986353 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 10:44:59.991394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:44:59.986428 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 10:45:00.974764 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:00.974722 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 10:45:00.975068 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:00.975030 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 10:45:01.977811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:01.977766 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 10:45:01.978456 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:01.978422 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 10:45:02.391433 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:02.391383 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 10:45:02.726394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:02.721443 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 10:45:02.901763 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:02.901719 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Nov 03 10:45:02.976513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:02.976419 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 10:45:02.976768 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:02.976737 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 10:45:03.283508 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:03.283386 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 10:45:03.533814 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:03.533704 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 10:45:03.538980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:03.538950 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:45:03.819413 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:03.735548 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 10:45:03.820549 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:03.820420 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:45:03.830356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:03.830329 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:45:03.879275 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:03.879243 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:45:03.991042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:03.989676 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 10:45:03.991042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:03.989965 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 10:45:04.976035 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:04.975991 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 10:45:04.976459 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:04.976063 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 10:45:05.975139 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:05.975104 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 10:45:05.975329 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:05.975161 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 10:45:06.974618 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:06.974574 5086 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 10:45:06.975024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:06.974653 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 10:45:08.685243 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:08.685204 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:45:08.685917 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:08.685647 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:45:08.686760 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:08.686708 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 10:45:12.394221 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.394181 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72"] Nov 03 10:45:12.399718 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.399694 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.415009 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.414978 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72"] Nov 03 10:45:12.434433 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.434400 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95"] Nov 03 10:45:12.440991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.440965 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.454165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.454136 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95"] Nov 03 10:45:12.511769 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.511731 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.511769 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.511773 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zd6j\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-kube-api-access-2zd6j\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.511986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.511795 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.511986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.511862 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.511986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.511909 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.511986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.511945 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkpv7\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-kube-api-access-fkpv7\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.512133 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512018 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.512133 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512053 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b9885d-8b0e-4e0b-9628-30034fb60dda-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.512133 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512090 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.512133 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512118 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.512257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512154 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.512257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512172 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c586beae-5009-458b-8704-e8bc2f21f2a2-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.512257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512188 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.512366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512320 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.512397 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512365 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.512430 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512404 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.512473 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512458 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.512507 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.512486 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.613668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.613625 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.613867 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.613677 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.613867 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.613747 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.613867 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.613774 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zd6j\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-kube-api-access-2zd6j\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.613867 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.613809 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.613867 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.613832 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.614135 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.613865 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.614135 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.613912 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkpv7\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-kube-api-access-fkpv7\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.614135 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.613984 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.614135 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614019 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b9885d-8b0e-4e0b-9628-30034fb60dda-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.614135 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614036 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614159 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614219 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614266 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614273 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614314 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614341 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c586beae-5009-458b-8704-e8bc2f21f2a2-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614370 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614400 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614433 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614481 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.614517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614505 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.615088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614650 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.615088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614818 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.615088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614866 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.615088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.614965 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b9885d-8b0e-4e0b-9628-30034fb60dda-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.615088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.615070 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c586beae-5009-458b-8704-e8bc2f21f2a2-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.615381 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.615073 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.616518 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.616492 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.616633 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.616527 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.617215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.617189 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.617346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.617214 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.622049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.622020 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.623470 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.623443 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.623571 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.623549 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkpv7\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-kube-api-access-fkpv7\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:12.623628 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.623580 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zd6j\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-kube-api-access-2zd6j\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.717479 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.717397 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:12.764394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:12.764363 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:13.011826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.011795 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72"] Nov 03 10:45:13.012890 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:45:13.012858 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b9885d_8b0e_4e0b_9628_30034fb60dda.slice/crio-254b7b28a6a43d9b726faa49afec376445dc915bce027c770a87cad4ded0ba3b WatchSource:0}: Error finding container 254b7b28a6a43d9b726faa49afec376445dc915bce027c770a87cad4ded0ba3b: Status 404 returned error can't find the container with id 254b7b28a6a43d9b726faa49afec376445dc915bce027c770a87cad4ded0ba3b Nov 03 10:45:13.015765 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.015731 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:45:13.015870 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.015801 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:45:13.015870 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.015833 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:45:13.073662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.073632 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95"] Nov 03 10:45:13.074454 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:45:13.074428 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc586beae_5009_458b_8704_e8bc2f21f2a2.slice/crio-5b606b25db98034fde034c2327f778cfad585c155379c34bfbc78454edae3fe1 WatchSource:0}: Error finding container 5b606b25db98034fde034c2327f778cfad585c155379c34bfbc78454edae3fe1: Status 404 returned error can't find the container with id 5b606b25db98034fde034c2327f778cfad585c155379c34bfbc78454edae3fe1 Nov 03 10:45:13.077265 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.077184 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:45:13.077265 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.077252 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:45:13.077453 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.077282 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:45:13.085988 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.085959 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" event={"ID":"44b9885d-8b0e-4e0b-9628-30034fb60dda","Type":"ContainerStarted","Data":"bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7"} Nov 03 10:45:13.086113 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.086022 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" event={"ID":"44b9885d-8b0e-4e0b-9628-30034fb60dda","Type":"ContainerStarted","Data":"254b7b28a6a43d9b726faa49afec376445dc915bce027c770a87cad4ded0ba3b"} Nov 03 10:45:13.087527 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.087500 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" event={"ID":"c586beae-5009-458b-8704-e8bc2f21f2a2","Type":"ContainerStarted","Data":"5b606b25db98034fde034c2327f778cfad585c155379c34bfbc78454edae3fe1"} Nov 03 10:45:13.103720 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.103661 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podStartSLOduration=1.103641564 podStartE2EDuration="1.103641564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:13.103436351 +0000 UTC m=+991.088307659" watchObservedRunningTime="2026-11-03 10:45:13.103641564 +0000 UTC m=+991.088512847" Nov 03 10:45:13.721376 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:13.719403 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:14.095489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:14.095449 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" event={"ID":"c586beae-5009-458b-8704-e8bc2f21f2a2","Type":"ContainerStarted","Data":"faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9"} Nov 03 10:45:14.149320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:14.138183 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podStartSLOduration=2.13816244 podStartE2EDuration="2.13816244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:14.136491285 +0000 UTC m=+992.121362575" watchObservedRunningTime="2026-11-03 10:45:14.13816244 +0000 UTC m=+992.123033727" Nov 03 10:45:14.721863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:14.721817 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 10:45:14.727433 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:14.727379 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 10:45:14.765873 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:14.765834 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:14.767550 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:14.767517 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 10:45:14.767662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:14.767575 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 10:45:15.719774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:15.719732 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 10:45:15.720114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:15.720060 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 10:45:15.765847 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:15.765808 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 10:45:15.766484 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:15.766450 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 10:45:16.719775 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:16.719738 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 10:45:16.720093 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:16.720060 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 10:45:16.766955 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:16.766917 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 10:45:16.767549 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:16.767517 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 10:45:17.737757 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:17.737716 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 10:45:17.738019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:17.737990 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 10:45:17.764968 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:17.764928 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 10:45:17.765267 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:17.765239 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 10:45:18.721780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:18.721735 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 10:45:18.722436 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:18.722401 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 10:45:18.772756 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:18.772712 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 10:45:18.774470 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:18.774434 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 10:45:19.734736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:19.734696 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 10:45:19.735382 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:19.735349 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 10:45:19.770320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:19.769741 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 10:45:19.770320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:19.769806 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 10:45:20.730819 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:20.730775 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 10:45:20.731112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:20.731076 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 10:45:20.841326 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:20.840505 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 10:45:20.841326 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:20.840576 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 10:45:21.848320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:21.847415 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 10:45:22.792908 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:22.792874 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:45:22.793159 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:22.793131 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:45:22.806826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:22.806792 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:45:22.807092 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:22.807063 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:45:23.223322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:23.215572 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:23.223322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:23.219900 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:23.265241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:23.264786 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:45:23.834535 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:23.834490 5086 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:45:23.834735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:23.834583 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:45:24.311701 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:45:24.311606 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:45:24.566397 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:24.565242 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:24.570047 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:24.570021 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:24.576132 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:24.573699 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 10:45:29.810309 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.810268 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq"] Nov 03 10:45:29.816479 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.816449 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.821637 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.821612 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f78621-openshift-default-dockercfg-2x24f\"" Nov 03 10:45:29.828219 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.828187 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq"] Nov 03 10:45:29.849778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.849744 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.849778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.849779 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.850019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.849901 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.850019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.849937 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.850134 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.850092 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.850193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.850137 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqx55\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-kube-api-access-kqx55\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.850249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.850202 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.850320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.850253 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.850381 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.850342 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.951437 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951393 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.951655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951490 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.951655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951516 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqx55\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-kube-api-access-kqx55\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.951655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951545 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.951655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951579 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.951655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951634 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.951926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951814 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.951926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951819 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.951926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951846 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.952087 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951925 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.952087 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.951959 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.952087 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.952032 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.952226 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.952092 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.952572 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.952554 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.954082 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.954056 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.954276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.954257 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.960939 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.960911 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:29.961140 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:29.961118 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqx55\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-kube-api-access-kqx55\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:30.138097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:30.138063 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:30.434357 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:30.434323 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq"] Nov 03 10:45:30.434510 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:45:30.434414 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a7f5f3_2a0f_4f34_aaa1_8dc03b2dbc56.slice/crio-50c73dd88d381c9ddda4c05fdc255ac13fe8d435d26c065b7aaf22d4c6ee67cd WatchSource:0}: Error finding container 50c73dd88d381c9ddda4c05fdc255ac13fe8d435d26c065b7aaf22d4c6ee67cd: Status 404 returned error can't find the container with id 50c73dd88d381c9ddda4c05fdc255ac13fe8d435d26c065b7aaf22d4c6ee67cd Nov 03 10:45:30.437231 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:30.437199 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:45:30.437356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:30.437271 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:45:30.437356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:30.437328 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:45:30.710359 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:30.710236 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" event={"ID":"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56","Type":"ContainerStarted","Data":"d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400"} Nov 03 10:45:30.710359 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:30.710313 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" event={"ID":"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56","Type":"ContainerStarted","Data":"50c73dd88d381c9ddda4c05fdc255ac13fe8d435d26c065b7aaf22d4c6ee67cd"} Nov 03 10:45:30.730506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:30.730448 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podStartSLOduration=1.730428488 podStartE2EDuration="1.730428488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:30.728092393 +0000 UTC m=+1008.712963677" watchObservedRunningTime="2026-11-03 10:45:30.730428488 +0000 UTC m=+1008.715299774" Nov 03 10:45:31.141690 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:31.141649 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:31.149770 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:31.147427 5086 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 10:45:31.149770 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:31.147492 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 10:45:32.141172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:32.140693 5086 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 10:45:32.141172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:32.140756 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 10:45:33.142926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:33.142888 5086 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 10:45:33.145597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:33.142954 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 10:45:34.145506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:34.145457 5086 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 10:45:34.146026 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:34.145777 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 10:45:35.141788 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.141750 5086 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 10:45:35.142112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.142067 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 10:45:35.193631 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.193596 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97"] Nov 03 10:45:35.198038 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.198018 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:35.205548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.205175 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ab592834-dockercfg-dmszl\"" Nov 03 10:45:35.205548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.205280 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ab592834\"" Nov 03 10:45:35.205548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.205282 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97"] Nov 03 10:45:35.255944 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.255913 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h"] Nov 03 10:45:35.261460 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.261418 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h"] Nov 03 10:45:35.261617 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.261533 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:35.285774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.285741 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqsjp\" (UniqueName: \"kubernetes.io/projected/a5754800-e23d-42ab-b135-12ff97a2eadf-kube-api-access-bqsjp\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97\" (UID: \"a5754800-e23d-42ab-b135-12ff97a2eadf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:35.285917 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.285900 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2d6x\" (UniqueName: \"kubernetes.io/projected/01b0e25b-1d36-4315-8a1a-d6df9df66a11-kube-api-access-p2d6x\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h\" (UID: \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:35.286040 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.286018 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5754800-e23d-42ab-b135-12ff97a2eadf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97\" (UID: \"a5754800-e23d-42ab-b135-12ff97a2eadf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:35.286195 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.286178 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b0e25b-1d36-4315-8a1a-d6df9df66a11-plugins-config-volume\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h\" (UID: \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:35.390544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.388025 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2d6x\" (UniqueName: \"kubernetes.io/projected/01b0e25b-1d36-4315-8a1a-d6df9df66a11-kube-api-access-p2d6x\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h\" (UID: \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:35.390544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.388156 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5754800-e23d-42ab-b135-12ff97a2eadf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97\" (UID: \"a5754800-e23d-42ab-b135-12ff97a2eadf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:35.390544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.388283 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b0e25b-1d36-4315-8a1a-d6df9df66a11-plugins-config-volume\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h\" (UID: \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:35.390544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.388402 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqsjp\" (UniqueName: \"kubernetes.io/projected/a5754800-e23d-42ab-b135-12ff97a2eadf-kube-api-access-bqsjp\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97\" (UID: \"a5754800-e23d-42ab-b135-12ff97a2eadf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:35.390544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.390376 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5754800-e23d-42ab-b135-12ff97a2eadf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97\" (UID: \"a5754800-e23d-42ab-b135-12ff97a2eadf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:35.392861 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.391168 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b0e25b-1d36-4315-8a1a-d6df9df66a11-plugins-config-volume\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h\" (UID: \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:35.398563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.397938 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqsjp\" (UniqueName: \"kubernetes.io/projected/a5754800-e23d-42ab-b135-12ff97a2eadf-kube-api-access-bqsjp\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97\" (UID: \"a5754800-e23d-42ab-b135-12ff97a2eadf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:35.401353 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.400532 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2d6x\" (UniqueName: \"kubernetes.io/projected/01b0e25b-1d36-4315-8a1a-d6df9df66a11-kube-api-access-p2d6x\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h\" (UID: \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:35.555325 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.552580 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:35.576894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.576865 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:35.888789 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:45:35.888756 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5754800_e23d_42ab_b135_12ff97a2eadf.slice/crio-c98b1dbc7337c39f15c62eaa24cf4ab70d9796073fff9619395646a4f5227a4b WatchSource:0}: Error finding container c98b1dbc7337c39f15c62eaa24cf4ab70d9796073fff9619395646a4f5227a4b: Status 404 returned error can't find the container with id c98b1dbc7337c39f15c62eaa24cf4ab70d9796073fff9619395646a4f5227a4b Nov 03 10:45:35.889093 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.889069 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97"] Nov 03 10:45:35.957499 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:35.957461 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h"] Nov 03 10:45:36.139179 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.138708 5086 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 10:45:36.139179 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.138776 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 10:45:36.814523 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.814481 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" event={"ID":"a5754800-e23d-42ab-b135-12ff97a2eadf","Type":"ContainerStarted","Data":"81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17"} Nov 03 10:45:36.814962 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.814689 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:36.814962 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.814724 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" event={"ID":"a5754800-e23d-42ab-b135-12ff97a2eadf","Type":"ContainerStarted","Data":"c98b1dbc7337c39f15c62eaa24cf4ab70d9796073fff9619395646a4f5227a4b"} Nov 03 10:45:36.816204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.816176 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" event={"ID":"01b0e25b-1d36-4315-8a1a-d6df9df66a11","Type":"ContainerStarted","Data":"37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0"} Nov 03 10:45:36.816363 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.816213 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" event={"ID":"01b0e25b-1d36-4315-8a1a-d6df9df66a11","Type":"ContainerStarted","Data":"09382310d743b7587493ecf3800ac5fd28365cd8b72c13cfd90a16597c723e72"} Nov 03 10:45:36.816363 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.816252 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:36.834246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.834191 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" podStartSLOduration=1.8341672199999999 podStartE2EDuration="1.83416722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:36.831157024 +0000 UTC m=+1014.816028310" watchObservedRunningTime="2026-11-03 10:45:36.83416722 +0000 UTC m=+1014.819038503" Nov 03 10:45:36.850433 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:36.850357 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" podStartSLOduration=1.850340692 podStartE2EDuration="1.850340692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:36.849735597 +0000 UTC m=+1014.834606938" watchObservedRunningTime="2026-11-03 10:45:36.850340692 +0000 UTC m=+1014.835211975" Nov 03 10:45:37.138785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:37.138735 5086 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 10:45:37.139020 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:37.138830 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 10:45:39.155821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.155776 5086 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:45:39.156358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.155839 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:45:39.201943 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.201908 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:39.203284 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.203256 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:39.204208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.204187 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 10:45:39.636920 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.636880 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd"] Nov 03 10:45:39.643396 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.643371 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.659467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.652400 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ab592834-serving-cert\"" Nov 03 10:45:39.659467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.653861 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd"] Nov 03 10:45:39.802753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.802190 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8fz\" (UniqueName: \"kubernetes.io/projected/3171b0ed-e03b-4466-a398-9568f50cb92b-kube-api-access-zk8fz\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.802753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.802418 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-maas-api-tls\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.802753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.802528 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-metrics-certs\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.802753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.802684 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3171b0ed-e03b-4466-a398-9568f50cb92b-openshift-service-ca\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.903787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.903706 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3171b0ed-e03b-4466-a398-9568f50cb92b-openshift-service-ca\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.903787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.903760 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8fz\" (UniqueName: \"kubernetes.io/projected/3171b0ed-e03b-4466-a398-9568f50cb92b-kube-api-access-zk8fz\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.903975 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.903910 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-maas-api-tls\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.903975 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.903942 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-metrics-certs\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.904545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.904522 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3171b0ed-e03b-4466-a398-9568f50cb92b-openshift-service-ca\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.906434 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.906411 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-maas-api-tls\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.906669 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.906653 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-metrics-certs\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.912818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.912797 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8fz\" (UniqueName: \"kubernetes.io/projected/3171b0ed-e03b-4466-a398-9568f50cb92b-kube-api-access-zk8fz\") pod \"maas-api-e2e-mt-ab592834-8448b97db-b88hd\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:39.966646 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:39.966616 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:40.249490 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:40.249456 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd"] Nov 03 10:45:40.250783 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:45:40.250759 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3171b0ed_e03b_4466_a398_9568f50cb92b.slice/crio-5a08fe8b2ed5bc839a3495c14fb960155547008db9977d016ba56cfea2f5000b WatchSource:0}: Error finding container 5a08fe8b2ed5bc839a3495c14fb960155547008db9977d016ba56cfea2f5000b: Status 404 returned error can't find the container with id 5a08fe8b2ed5bc839a3495c14fb960155547008db9977d016ba56cfea2f5000b Nov 03 10:45:40.252681 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:40.252656 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:45:40.859774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:40.859721 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" event={"ID":"3171b0ed-e03b-4466-a398-9568f50cb92b","Type":"ContainerStarted","Data":"39ae33e7ce184812b6571c6c776c12796d8a9f5098cb3724b45dcddeb8234d5a"} Nov 03 10:45:40.859964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:40.859841 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" event={"ID":"3171b0ed-e03b-4466-a398-9568f50cb92b","Type":"ContainerStarted","Data":"5a08fe8b2ed5bc839a3495c14fb960155547008db9977d016ba56cfea2f5000b"} Nov 03 10:45:40.859964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:40.859879 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:40.878773 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:40.878700 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" podStartSLOduration=1.398481069 podStartE2EDuration="1.878686284s" totalImagesPullingTime="480.205215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:40.876915686 +0000 UTC m=+1018.861786969" watchObservedRunningTime="2026-11-03 10:45:40.878686284 +0000 UTC m=+1018.863557588" Nov 03 10:45:43.914322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:43.911443 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 10:45:44.945322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:44.912449 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 10:45:44.945322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:44.913126 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:45:45.855331 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:45.842862 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:45:46.895724 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:46.895694 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:45:47.828990 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:47.828963 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:45:47.829188 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:47.829044 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:45:50.359358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.359321 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2"] Nov 03 10:45:50.378636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.378607 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2"] Nov 03 10:45:50.378875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.378777 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.382446 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.382424 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-141053-serving-cert\"" Nov 03 10:45:50.404516 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.404484 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-metrics-certs\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.404649 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.404550 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstdw\" (UniqueName: \"kubernetes.io/projected/837be16e-5f29-45fb-91c3-651647739962-kube-api-access-hstdw\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.404774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.404746 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.404985 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.404957 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837be16e-5f29-45fb-91c3-651647739962-openshift-service-ca\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.518082 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.518037 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837be16e-5f29-45fb-91c3-651647739962-openshift-service-ca\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.518336 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.518179 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-metrics-certs\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.518336 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.518234 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstdw\" (UniqueName: \"kubernetes.io/projected/837be16e-5f29-45fb-91c3-651647739962-kube-api-access-hstdw\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.519332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.518372 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.519332 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:45:50.518646 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-141053-serving-cert: secret "maas-api-e2e-api-a-141053-serving-cert" not found Nov 03 10:45:50.519332 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:45:50.518989 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls podName:837be16e-5f29-45fb-91c3-651647739962 nodeName:}" failed. No retries permitted until 2026-11-03 10:45:51.018960851 +0000 UTC m=+1029.003832133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls") pod "maas-api-e2e-api-a-141053-55478bddb5-cbnb2" (UID: "837be16e-5f29-45fb-91c3-651647739962") : secret "maas-api-e2e-api-a-141053-serving-cert" not found Nov 03 10:45:50.520492 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.520469 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837be16e-5f29-45fb-91c3-651647739962-openshift-service-ca\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.526679 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.526642 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-metrics-certs\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:50.534578 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:50.534551 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstdw\" (UniqueName: \"kubernetes.io/projected/837be16e-5f29-45fb-91c3-651647739962-kube-api-access-hstdw\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:51.025868 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:51.025827 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:51.029492 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:51.029466 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls\") pod \"maas-api-e2e-api-a-141053-55478bddb5-cbnb2\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:51.298029 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:51.297937 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:51.580757 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:51.580723 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2"] Nov 03 10:45:51.581921 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:45:51.581888 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837be16e_5f29_45fb_91c3_651647739962.slice/crio-2b5ea4dc0181667fd84c8d632fbd6a2d5385a86a135cdee6ea1bc405b7498962 WatchSource:0}: Error finding container 2b5ea4dc0181667fd84c8d632fbd6a2d5385a86a135cdee6ea1bc405b7498962: Status 404 returned error can't find the container with id 2b5ea4dc0181667fd84c8d632fbd6a2d5385a86a135cdee6ea1bc405b7498962 Nov 03 10:45:51.891315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:51.891219 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" event={"ID":"837be16e-5f29-45fb-91c3-651647739962","Type":"ContainerStarted","Data":"2b5ea4dc0181667fd84c8d632fbd6a2d5385a86a135cdee6ea1bc405b7498962"} Nov 03 10:45:52.899222 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:52.899137 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" event={"ID":"837be16e-5f29-45fb-91c3-651647739962","Type":"ContainerStarted","Data":"c51486ba5dc06926c7a01986117674c298c08ad66a0adb31481112caf72bcdf3"} Nov 03 10:45:52.899673 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:52.899233 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:45:52.914833 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:52.914788 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" podStartSLOduration=2.386251943 podStartE2EDuration="2.914776632s" totalImagesPullingTime="528.524689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:52.914439714 +0000 UTC m=+1030.899310997" watchObservedRunningTime="2026-11-03 10:45:52.914776632 +0000 UTC m=+1030.899647915" Nov 03 10:45:54.293575 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:45:54.293513 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:45:58.911862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:45:58.911832 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:46:01.896322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:01.894510 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 10:46:02.038316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:02.036428 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:46:02.866866 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:02.866820 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Nov 03 10:46:02.914768 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:02.912387 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 10:46:02.914768 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:02.913217 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:46:03.093319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:03.092860 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 10:46:03.110325 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:03.093680 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:46:03.957690 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:03.950393 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 10:46:04.000325 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:03.997442 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 10:46:04.061239 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:04.061205 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 10:46:14.191807 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.191773 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6"] Nov 03 10:46:14.198233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.198209 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:14.202281 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.202252 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-151744\"" Nov 03 10:46:14.203631 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.203606 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6"] Nov 03 10:46:14.204531 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.204497 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-151744-dockercfg-h689g\"" Nov 03 10:46:14.248260 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.248220 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5036ae98-bc81-458b-a5d4-24f31c86c422-plugins-config-volume\") pod \"payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6\" (UID: \"5036ae98-bc81-458b-a5d4-24f31c86c422\") " pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:14.248472 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.248441 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8slb\" (UniqueName: \"kubernetes.io/projected/5036ae98-bc81-458b-a5d4-24f31c86c422-kube-api-access-s8slb\") pod \"payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6\" (UID: \"5036ae98-bc81-458b-a5d4-24f31c86c422\") " pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:14.352317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.351382 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8slb\" (UniqueName: \"kubernetes.io/projected/5036ae98-bc81-458b-a5d4-24f31c86c422-kube-api-access-s8slb\") pod \"payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6\" (UID: \"5036ae98-bc81-458b-a5d4-24f31c86c422\") " pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:14.352317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.351651 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5036ae98-bc81-458b-a5d4-24f31c86c422-plugins-config-volume\") pod \"payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6\" (UID: \"5036ae98-bc81-458b-a5d4-24f31c86c422\") " pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:14.352617 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.352590 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5036ae98-bc81-458b-a5d4-24f31c86c422-plugins-config-volume\") pod \"payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6\" (UID: \"5036ae98-bc81-458b-a5d4-24f31c86c422\") " pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:14.426366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.425926 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8slb\" (UniqueName: \"kubernetes.io/projected/5036ae98-bc81-458b-a5d4-24f31c86c422-kube-api-access-s8slb\") pod \"payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6\" (UID: \"5036ae98-bc81-458b-a5d4-24f31c86c422\") " pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:14.519941 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.519531 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:14.818272 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:14.818246 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6"] Nov 03 10:46:14.819913 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:46:14.819878 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5036ae98_bc81_458b_a5d4_24f31c86c422.slice/crio-4516d08d78096e454e3fd7e1b8d1dfa4b4cea4b4ce7a717f693c29e6eafc7f6c WatchSource:0}: Error finding container 4516d08d78096e454e3fd7e1b8d1dfa4b4cea4b4ce7a717f693c29e6eafc7f6c: Status 404 returned error can't find the container with id 4516d08d78096e454e3fd7e1b8d1dfa4b4cea4b4ce7a717f693c29e6eafc7f6c Nov 03 10:46:15.202773 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.202692 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8"] Nov 03 10:46:15.207487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.207462 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.221980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.221953 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8"] Nov 03 10:46:15.269926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.269896 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.270089 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.269952 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.270204 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.270182 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.270269 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.270224 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.270362 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.270346 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.270427 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.270396 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe9657-ae56-48e9-b18d-096c0180cf28-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.270506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.270487 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.270544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.270521 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czgzn\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-kube-api-access-czgzn\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.270687 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.270671 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.301340 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.301313 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ab592834-openshift-default-dockercfg-vq7pv\"" Nov 03 10:46:15.372425 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372391 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.372596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372446 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe9657-ae56-48e9-b18d-096c0180cf28-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.372596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372488 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.372596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372508 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czgzn\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-kube-api-access-czgzn\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.372596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372577 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.372835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372649 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.372835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372693 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.372835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372794 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.372835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372823 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.373044 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.372832 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.373212 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.373188 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.373381 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.373286 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe9657-ae56-48e9-b18d-096c0180cf28-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.373530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.373507 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.373668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.373518 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.375322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.375288 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.375416 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.375390 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.381936 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.381910 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czgzn\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-kube-api-access-czgzn\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.382023 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.381951 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.537614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.537516 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:15.601233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.601194 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" event={"ID":"5036ae98-bc81-458b-a5d4-24f31c86c422","Type":"ContainerStarted","Data":"2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9"} Nov 03 10:46:15.601425 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.601264 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" event={"ID":"5036ae98-bc81-458b-a5d4-24f31c86c422","Type":"ContainerStarted","Data":"4516d08d78096e454e3fd7e1b8d1dfa4b4cea4b4ce7a717f693c29e6eafc7f6c"} Nov 03 10:46:15.601581 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.601553 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:15.621032 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.620977 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" podStartSLOduration=1.620962212 podStartE2EDuration="1.620962212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:15.617727118 +0000 UTC m=+1053.602598403" watchObservedRunningTime="2026-11-03 10:46:15.620962212 +0000 UTC m=+1053.605833494" Nov 03 10:46:15.851248 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.851051 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8"] Nov 03 10:46:15.855027 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:46:15.854995 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafe9657_ae56_48e9_b18d_096c0180cf28.slice/crio-a19eb83a05f839e0db401d9304d19469b6c285f076a335ae3eaf75758613a8ba WatchSource:0}: Error finding container a19eb83a05f839e0db401d9304d19469b6c285f076a335ae3eaf75758613a8ba: Status 404 returned error can't find the container with id a19eb83a05f839e0db401d9304d19469b6c285f076a335ae3eaf75758613a8ba Nov 03 10:46:15.858034 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.857999 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:46:15.858125 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.858069 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:46:15.858125 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:15.858102 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:46:16.609590 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:16.609558 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" event={"ID":"aafe9657-ae56-48e9-b18d-096c0180cf28","Type":"ContainerStarted","Data":"e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b"} Nov 03 10:46:16.610024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:16.609611 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" event={"ID":"aafe9657-ae56-48e9-b18d-096c0180cf28","Type":"ContainerStarted","Data":"a19eb83a05f839e0db401d9304d19469b6c285f076a335ae3eaf75758613a8ba"} Nov 03 10:46:16.640444 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:16.640357 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" podStartSLOduration=1.6403328350000002 podStartE2EDuration="1.640332835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:16.628454931 +0000 UTC m=+1054.613326215" watchObservedRunningTime="2026-11-03 10:46:16.640332835 +0000 UTC m=+1054.625204118" Nov 03 10:46:17.541482 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:17.541393 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:17.544457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:17.544427 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 10:46:17.544716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:17.544689 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 10:46:18.541187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:18.540731 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 10:46:18.541187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:18.540796 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 10:46:19.538682 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:19.538645 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 10:46:19.538962 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:19.538933 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 10:46:19.884045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:19.884015 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q"] Nov 03 10:46:19.889810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:19.889784 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:19.896761 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:19.896738 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-151744-serving-cert\"" Nov 03 10:46:19.897517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:19.897487 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q"] Nov 03 10:46:20.059316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.058964 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316eabbc-0694-4267-8894-47cd1b6fd7ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.059316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.059025 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwmjm\" (UniqueName: \"kubernetes.io/projected/316eabbc-0694-4267-8894-47cd1b6fd7ea-kube-api-access-qwmjm\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.059316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.059144 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-maas-api-tls\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.059316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.059225 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-metrics-certs\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.160929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.160840 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316eabbc-0694-4267-8894-47cd1b6fd7ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.160929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.160883 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwmjm\" (UniqueName: \"kubernetes.io/projected/316eabbc-0694-4267-8894-47cd1b6fd7ea-kube-api-access-qwmjm\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.161187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.160971 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-maas-api-tls\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.161187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.161054 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-metrics-certs\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.161777 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.161746 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316eabbc-0694-4267-8894-47cd1b6fd7ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.165981 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.165952 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-maas-api-tls\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.166447 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.166424 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-metrics-certs\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.168983 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.168960 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwmjm\" (UniqueName: \"kubernetes.io/projected/316eabbc-0694-4267-8894-47cd1b6fd7ea-kube-api-access-qwmjm\") pod \"maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.247783 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.247751 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:20.539370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.538687 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 10:46:20.539370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.538751 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 10:46:20.549683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.549652 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q"] Nov 03 10:46:20.556869 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:46:20.556560 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316eabbc_0694_4267_8894_47cd1b6fd7ea.slice/crio-ee3e50e81e11b838cffd5555c9e25f80dc593cc238b812f73e142d0930cea81e WatchSource:0}: Error finding container ee3e50e81e11b838cffd5555c9e25f80dc593cc238b812f73e142d0930cea81e: Status 404 returned error can't find the container with id ee3e50e81e11b838cffd5555c9e25f80dc593cc238b812f73e142d0930cea81e Nov 03 10:46:20.669290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:20.669243 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" event={"ID":"316eabbc-0694-4267-8894-47cd1b6fd7ea","Type":"ContainerStarted","Data":"ee3e50e81e11b838cffd5555c9e25f80dc593cc238b812f73e142d0930cea81e"} Nov 03 10:46:21.541231 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:21.541176 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 10:46:21.541903 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:21.541843 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 10:46:21.678512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:21.678476 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" event={"ID":"316eabbc-0694-4267-8894-47cd1b6fd7ea","Type":"ContainerStarted","Data":"64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e"} Nov 03 10:46:21.678701 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:21.678679 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:21.697239 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:21.697185 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" podStartSLOduration=2.206904647 podStartE2EDuration="2.697171961s" totalImagesPullingTime="490.267314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:21.695432896 +0000 UTC m=+1059.680304180" watchObservedRunningTime="2026-11-03 10:46:21.697171961 +0000 UTC m=+1059.682043259" Nov 03 10:46:22.542629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:22.542599 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:22.543089 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:22.543070 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:22.543877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:22.543857 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:46:24.277676 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:46:24.277642 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:46:26.613033 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:26.612995 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:46:27.691322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:27.691264 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:46:32.954694 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:32.954659 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql"] Nov 03 10:46:32.959587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:32.959568 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:32.972330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:32.972278 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql"] Nov 03 10:46:33.002834 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.002064 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd442a16-582f-4de7-b474-fee20cdc185f-openshift-service-ca\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.003167 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.003134 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-maas-api-tls\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.003283 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.003204 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-metrics-certs\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.003420 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.003381 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzbt\" (UniqueName: \"kubernetes.io/projected/dd442a16-582f-4de7-b474-fee20cdc185f-kube-api-access-9nzbt\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.005967 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.005948 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a917b1-serving-cert\"" Nov 03 10:46:33.104875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.104842 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzbt\" (UniqueName: \"kubernetes.io/projected/dd442a16-582f-4de7-b474-fee20cdc185f-kube-api-access-9nzbt\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.105144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.104935 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd442a16-582f-4de7-b474-fee20cdc185f-openshift-service-ca\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.105144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.104984 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-maas-api-tls\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.105144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.105015 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-metrics-certs\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.105705 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.105678 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd442a16-582f-4de7-b474-fee20cdc185f-openshift-service-ca\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.107591 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.107558 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-metrics-certs\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.107712 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.107623 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-maas-api-tls\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.113606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.113584 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzbt\" (UniqueName: \"kubernetes.io/projected/dd442a16-582f-4de7-b474-fee20cdc185f-kube-api-access-9nzbt\") pod \"maas-api-e2e-api-b-a917b1-864b97db97-7x8ql\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.281716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.281639 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:33.576110 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.576070 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql"] Nov 03 10:46:33.578016 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:46:33.577988 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd442a16_582f_4de7_b474_fee20cdc185f.slice/crio-4b35af4b6a43eb53a64fddd63f49fcab9f4b4e32a21fc09a003d3deb7905ddaf WatchSource:0}: Error finding container 4b35af4b6a43eb53a64fddd63f49fcab9f4b4e32a21fc09a003d3deb7905ddaf: Status 404 returned error can't find the container with id 4b35af4b6a43eb53a64fddd63f49fcab9f4b4e32a21fc09a003d3deb7905ddaf Nov 03 10:46:33.871526 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:33.871445 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" event={"ID":"dd442a16-582f-4de7-b474-fee20cdc185f","Type":"ContainerStarted","Data":"4b35af4b6a43eb53a64fddd63f49fcab9f4b4e32a21fc09a003d3deb7905ddaf"} Nov 03 10:46:34.890246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:34.890209 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" event={"ID":"dd442a16-582f-4de7-b474-fee20cdc185f","Type":"ContainerStarted","Data":"668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e"} Nov 03 10:46:34.890734 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:34.890555 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:34.913103 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:34.913048 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" podStartSLOduration=2.474282632 podStartE2EDuration="2.913030526s" totalImagesPullingTime="438.747894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:34.908021432 +0000 UTC m=+1072.892892719" watchObservedRunningTime="2026-11-03 10:46:34.913030526 +0000 UTC m=+1072.897901809" Nov 03 10:46:38.675771 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.675737 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t"] Nov 03 10:46:38.680636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.680609 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:38.690950 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.690919 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7d8392\"" Nov 03 10:46:38.694530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.694497 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t"] Nov 03 10:46:38.815376 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.815331 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5r94\" (UniqueName: \"kubernetes.io/projected/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-kube-api-access-l5r94\") pod \"payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t\" (UID: \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:38.815596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.815491 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t\" (UID: \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:38.845863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.845825 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk"] Nov 03 10:46:38.895375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.895337 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk"] Nov 03 10:46:38.923095 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.900471 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:38.951916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.930739 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7d8392-dockercfg-78dkz\"" Nov 03 10:46:38.956921 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.956885 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5r94\" (UniqueName: \"kubernetes.io/projected/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-kube-api-access-l5r94\") pod \"payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t\" (UID: \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:38.957258 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.957239 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t\" (UID: \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:38.965131 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:38.958264 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t\" (UID: \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:39.006371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.005523 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5r94\" (UniqueName: \"kubernetes.io/projected/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-kube-api-access-l5r94\") pod \"payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t\" (UID: \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:39.059211 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.059174 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhvjn\" (UniqueName: \"kubernetes.io/projected/50243e18-189d-4f8d-a3a2-f382b90af7d6-kube-api-access-vhvjn\") pod \"payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk\" (UID: \"50243e18-189d-4f8d-a3a2-f382b90af7d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:39.059419 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.059288 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50243e18-189d-4f8d-a3a2-f382b90af7d6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk\" (UID: \"50243e18-189d-4f8d-a3a2-f382b90af7d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:39.160783 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.160748 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhvjn\" (UniqueName: \"kubernetes.io/projected/50243e18-189d-4f8d-a3a2-f382b90af7d6-kube-api-access-vhvjn\") pod \"payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk\" (UID: \"50243e18-189d-4f8d-a3a2-f382b90af7d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:39.161023 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.160805 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50243e18-189d-4f8d-a3a2-f382b90af7d6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk\" (UID: \"50243e18-189d-4f8d-a3a2-f382b90af7d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:39.161557 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.161535 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50243e18-189d-4f8d-a3a2-f382b90af7d6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk\" (UID: \"50243e18-189d-4f8d-a3a2-f382b90af7d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:39.168811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.168788 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhvjn\" (UniqueName: \"kubernetes.io/projected/50243e18-189d-4f8d-a3a2-f382b90af7d6-kube-api-access-vhvjn\") pod \"payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk\" (UID: \"50243e18-189d-4f8d-a3a2-f382b90af7d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:39.304096 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.304008 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:39.319304 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.319271 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:39.562968 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.562891 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx"] Nov 03 10:46:39.568418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.568396 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr"] Nov 03 10:46:39.568546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.568504 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:39.575780 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.574284 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:39.577421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.577394 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-273012\"" Nov 03 10:46:39.578642 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.578620 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx"] Nov 03 10:46:39.600894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.600873 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr"] Nov 03 10:46:39.605116 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:46:39.605092 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b25364_0fa4_4687_9c08_d48e2f9eba7d.slice/crio-30a66c7457d0c923d789c395a36f579aee46d220f6f9c5850c7a9fdfd8f3f9f6 WatchSource:0}: Error finding container 30a66c7457d0c923d789c395a36f579aee46d220f6f9c5850c7a9fdfd8f3f9f6: Status 404 returned error can't find the container with id 30a66c7457d0c923d789c395a36f579aee46d220f6f9c5850c7a9fdfd8f3f9f6 Nov 03 10:46:39.609170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.609146 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t"] Nov 03 10:46:39.629441 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:46:39.629417 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50243e18_189d_4f8d_a3a2_f382b90af7d6.slice/crio-28bc3038b3b2b4481e889c93ca85b842031a685ef71b38b417871130967fe851 WatchSource:0}: Error finding container 28bc3038b3b2b4481e889c93ca85b842031a685ef71b38b417871130967fe851: Status 404 returned error can't find the container with id 28bc3038b3b2b4481e889c93ca85b842031a685ef71b38b417871130967fe851 Nov 03 10:46:39.629692 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.629666 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk"] Nov 03 10:46:39.673992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.671101 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-plugins-config-volume\") pod \"payload-processing-e2e-api-b-273012-8665bbb447-4h8kx\" (UID: \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\") " pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:39.673992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.672390 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2d5j\" (UniqueName: \"kubernetes.io/projected/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-kube-api-access-t2d5j\") pod \"payload-processing-e2e-api-b-273012-8665bbb447-4h8kx\" (UID: \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\") " pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:39.687771 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.676160 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fccf\" (UniqueName: \"kubernetes.io/projected/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-kube-api-access-5fccf\") pod \"payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr\" (UID: \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:39.687771 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.676247 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr\" (UID: \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:39.819389 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.778540 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-plugins-config-volume\") pod \"payload-processing-e2e-api-b-273012-8665bbb447-4h8kx\" (UID: \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\") " pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:39.819389 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.778610 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2d5j\" (UniqueName: \"kubernetes.io/projected/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-kube-api-access-t2d5j\") pod \"payload-processing-e2e-api-b-273012-8665bbb447-4h8kx\" (UID: \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\") " pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:39.819389 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.778749 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fccf\" (UniqueName: \"kubernetes.io/projected/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-kube-api-access-5fccf\") pod \"payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr\" (UID: \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:39.819389 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.778812 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr\" (UID: \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:39.819389 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.779794 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr\" (UID: \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:39.832990 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.832926 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2d5j\" (UniqueName: \"kubernetes.io/projected/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-kube-api-access-t2d5j\") pod \"payload-processing-e2e-api-b-273012-8665bbb447-4h8kx\" (UID: \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\") " pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:39.839327 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.834809 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-plugins-config-volume\") pod \"payload-processing-e2e-api-b-273012-8665bbb447-4h8kx\" (UID: \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\") " pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:39.847925 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.847879 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fccf\" (UniqueName: \"kubernetes.io/projected/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-kube-api-access-5fccf\") pod \"payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr\" (UID: \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:39.892942 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.892904 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:39.904783 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:39.904754 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:40.016822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.016780 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" event={"ID":"50243e18-189d-4f8d-a3a2-f382b90af7d6","Type":"ContainerStarted","Data":"de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c"} Nov 03 10:46:40.016995 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.016866 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" event={"ID":"50243e18-189d-4f8d-a3a2-f382b90af7d6","Type":"ContainerStarted","Data":"28bc3038b3b2b4481e889c93ca85b842031a685ef71b38b417871130967fe851"} Nov 03 10:46:40.016995 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.016895 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:40.018911 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.018886 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" event={"ID":"c5b25364-0fa4-4687-9c08-d48e2f9eba7d","Type":"ContainerStarted","Data":"58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c"} Nov 03 10:46:40.019042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.018931 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" event={"ID":"c5b25364-0fa4-4687-9c08-d48e2f9eba7d","Type":"ContainerStarted","Data":"30a66c7457d0c923d789c395a36f579aee46d220f6f9c5850c7a9fdfd8f3f9f6"} Nov 03 10:46:40.019202 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.019185 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:40.036833 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.036779 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" podStartSLOduration=2.036761312 podStartE2EDuration="2.036761312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:40.034992122 +0000 UTC m=+1078.019863406" watchObservedRunningTime="2026-11-03 10:46:40.036761312 +0000 UTC m=+1078.021632598" Nov 03 10:46:40.053480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.053429 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" podStartSLOduration=2.053415655 podStartE2EDuration="2.053415655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:40.049692265 +0000 UTC m=+1078.034563549" watchObservedRunningTime="2026-11-03 10:46:40.053415655 +0000 UTC m=+1078.038286938" Nov 03 10:46:40.223562 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:46:40.220999 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f2dead_f4bf_4c3b_b1c8_0245dade0fde.slice/crio-31699a046eebd4ec77fe1d8f964b0ed531368d1f098382ba2cd29275731c3e82 WatchSource:0}: Error finding container 31699a046eebd4ec77fe1d8f964b0ed531368d1f098382ba2cd29275731c3e82: Status 404 returned error can't find the container with id 31699a046eebd4ec77fe1d8f964b0ed531368d1f098382ba2cd29275731c3e82 Nov 03 10:46:40.261716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.261686 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx"] Nov 03 10:46:40.264811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.264787 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr"] Nov 03 10:46:40.913828 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:40.913784 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" podUID="dd442a16-582f-4de7-b474-fee20cdc185f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.100:8443/health\": dial tcp 10.132.0.100:8443: connect: connection refused" Nov 03 10:46:41.027579 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:41.027542 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" event={"ID":"90f2dead-f4bf-4c3b-b1c8-0245dade0fde","Type":"ContainerStarted","Data":"9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4"} Nov 03 10:46:41.027579 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:41.027595 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" event={"ID":"90f2dead-f4bf-4c3b-b1c8-0245dade0fde","Type":"ContainerStarted","Data":"31699a046eebd4ec77fe1d8f964b0ed531368d1f098382ba2cd29275731c3e82"} Nov 03 10:46:41.027846 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:41.027739 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:41.029229 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:41.029205 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" event={"ID":"eb792017-70d8-4fc3-95e1-1cea7ef75cd0","Type":"ContainerStarted","Data":"f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171"} Nov 03 10:46:41.029392 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:41.029246 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:41.029392 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:41.029263 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" event={"ID":"eb792017-70d8-4fc3-95e1-1cea7ef75cd0","Type":"ContainerStarted","Data":"94cbb6f881d32c5017398a32f36eb684c5dbc1c1eb18fe0e73776823ea8ffd7e"} Nov 03 10:46:41.042989 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:41.042947 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" podStartSLOduration=2.042936027 podStartE2EDuration="2.042936027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:41.041922142 +0000 UTC m=+1079.026793424" watchObservedRunningTime="2026-11-03 10:46:41.042936027 +0000 UTC m=+1079.027807309" Nov 03 10:46:41.057043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:41.057001 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" podStartSLOduration=2.056990956 podStartE2EDuration="2.056990956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:41.055597247 +0000 UTC m=+1079.040468532" watchObservedRunningTime="2026-11-03 10:46:41.056990956 +0000 UTC m=+1079.041862238" Nov 03 10:46:43.100203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:43.100167 5086 scope.go:122] "RemoveContainer" containerID="ec42b5e73573d3d49b16d97b3e68987755302c5919db13181cb046a0c4656207" Nov 03 10:46:45.062501 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.062445 5086 generic.go:356] "Generic (PLEG): container finished" podID="dd442a16-582f-4de7-b474-fee20cdc185f" containerID="668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e" exitCode=1 Nov 03 10:46:45.062961 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.062534 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" event={"ID":"dd442a16-582f-4de7-b474-fee20cdc185f","Type":"ContainerDied","Data":"668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e"} Nov 03 10:46:45.063028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.063010 5086 scope.go:122] "RemoveContainer" containerID="668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e" Nov 03 10:46:45.335739 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.335695 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl"] Nov 03 10:46:45.340862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.340838 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.356276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.354256 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl"] Nov 03 10:46:45.388888 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.388844 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28wqw\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-kube-api-access-28wqw\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.389051 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.388968 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.389051 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.389012 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.389142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.389062 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.389219 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.389193 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.389454 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.389434 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db155cc-6494-41bc-8c24-b8bc6addfae6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.389514 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.389494 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.389548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.389529 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.389611 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.389597 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.501783 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.501742 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db155cc-6494-41bc-8c24-b8bc6addfae6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.502090 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.502067 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.502317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.502283 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.502455 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.502437 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.502611 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.502593 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28wqw\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-kube-api-access-28wqw\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.502726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.502714 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.502859 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.502832 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.503032 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.502908 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.503032 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.502950 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.504334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.504123 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db155cc-6494-41bc-8c24-b8bc6addfae6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.504501 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.504477 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.509455 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.509426 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.510044 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.510016 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.510199 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.510174 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.512448 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.512420 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.512694 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.512671 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.514205 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.513725 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.520836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.520812 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28wqw\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-kube-api-access-28wqw\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.662963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.662935 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:45.953058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.953027 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl"] Nov 03 10:46:45.954172 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:46:45.954133 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db155cc_6494_41bc_8c24_b8bc6addfae6.slice/crio-d58d16a3eabb3be20b3eff46e504dbbf2b97617757d9b2b574b80467e18c7944 WatchSource:0}: Error finding container d58d16a3eabb3be20b3eff46e504dbbf2b97617757d9b2b574b80467e18c7944: Status 404 returned error can't find the container with id d58d16a3eabb3be20b3eff46e504dbbf2b97617757d9b2b574b80467e18c7944 Nov 03 10:46:45.957080 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.957048 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:46:45.957158 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.957145 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:46:45.957206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:45.957182 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:46:46.075791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.075755 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" event={"ID":"6db155cc-6494-41bc-8c24-b8bc6addfae6","Type":"ContainerStarted","Data":"96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199"} Nov 03 10:46:46.076256 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.075836 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" event={"ID":"6db155cc-6494-41bc-8c24-b8bc6addfae6","Type":"ContainerStarted","Data":"d58d16a3eabb3be20b3eff46e504dbbf2b97617757d9b2b574b80467e18c7944"} Nov 03 10:46:46.079594 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.079565 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" event={"ID":"dd442a16-582f-4de7-b474-fee20cdc185f","Type":"ContainerStarted","Data":"edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0"} Nov 03 10:46:46.079813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.079792 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:46.232584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.232089 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf"] Nov 03 10:46:46.241440 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.240693 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.249642 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.249620 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-273012-serving-cert\"" Nov 03 10:46:46.265901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.265874 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf"] Nov 03 10:46:46.315368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.315320 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-maas-api-tls\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.315648 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.315616 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm4gp\" (UniqueName: \"kubernetes.io/projected/24126f15-928a-4852-a152-28584eca5081-kube-api-access-bm4gp\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.315739 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.315670 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-metrics-certs\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.315739 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.315714 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24126f15-928a-4852-a152-28584eca5081-openshift-service-ca\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.417473 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.417438 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-maas-api-tls\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.417716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.417615 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm4gp\" (UniqueName: \"kubernetes.io/projected/24126f15-928a-4852-a152-28584eca5081-kube-api-access-bm4gp\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.417716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.417655 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-metrics-certs\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.417716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.417686 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24126f15-928a-4852-a152-28584eca5081-openshift-service-ca\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.418453 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.418425 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24126f15-928a-4852-a152-28584eca5081-openshift-service-ca\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.420125 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.420101 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-metrics-certs\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.420222 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.420196 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-maas-api-tls\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.425228 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.425200 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm4gp\" (UniqueName: \"kubernetes.io/projected/24126f15-928a-4852-a152-28584eca5081-kube-api-access-bm4gp\") pod \"maas-api-e2e-api-b-273012-59b7568769-nb5vf\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.574540 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.574491 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:46.970348 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:46.965735 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf"] Nov 03 10:46:47.090580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:47.090534 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" event={"ID":"24126f15-928a-4852-a152-28584eca5081","Type":"ContainerStarted","Data":"fd359e394a76816719f321ace976bfc006f53897e5efdc96a1ffe3243f3be0e0"} Nov 03 10:46:47.129316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:47.128677 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" podStartSLOduration=2.128653482 podStartE2EDuration="2.128653482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:47.109086193 +0000 UTC m=+1085.093957549" watchObservedRunningTime="2026-11-03 10:46:47.128653482 +0000 UTC m=+1085.113524769" Nov 03 10:46:47.667987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:47.665810 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:47.669468 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:47.669434 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 10:46:47.669593 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:47.669493 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 10:46:48.099468 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:48.099430 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" event={"ID":"24126f15-928a-4852-a152-28584eca5081","Type":"ContainerStarted","Data":"43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1"} Nov 03 10:46:48.099856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:48.099711 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:48.137320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:48.136712 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" podStartSLOduration=1.517350161 podStartE2EDuration="2.13669177s" totalImagesPullingTime="619.341609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:48.120676249 +0000 UTC m=+1086.105547532" watchObservedRunningTime="2026-11-03 10:46:48.13669177 +0000 UTC m=+1086.121563055" Nov 03 10:46:48.677319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:48.664452 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 10:46:48.677319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:48.664577 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 10:46:49.663450 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:49.663416 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 10:46:49.664151 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:49.663474 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 10:46:50.664661 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:50.664458 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 10:46:50.664661 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:50.664527 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 10:46:51.042549 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:51.042465 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:46:51.043925 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:51.043901 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:46:51.663258 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:51.663224 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 10:46:51.663482 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:51.663310 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 10:46:52.049146 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:52.049018 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:46:52.050816 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:52.050794 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:46:52.100539 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:52.100508 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:46:52.664221 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:52.664183 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 10:46:52.664422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:52.664265 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 10:46:53.668856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:53.668830 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:53.669368 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:53.669348 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:53.669953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:53.669929 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:46:54.113753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:46:54.113723 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:46:54.279801 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:46:54.279757 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:47:00.305954 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.305918 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7"] Nov 03 10:47:00.310957 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.310935 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.324725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.324695 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7"] Nov 03 10:47:00.394358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.394312 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.394548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.394421 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.394548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.394471 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.394692 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.394584 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.394692 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.394603 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.394692 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.394622 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkblf\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-kube-api-access-rkblf\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.394859 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.394724 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.394859 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.394760 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.394859 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.394778 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496004 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.495969 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496028 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496048 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496144 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496365 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496340 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496414 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496526 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496511 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496593 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496530 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496593 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496540 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496697 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496593 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkblf\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-kube-api-access-rkblf\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496697 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496630 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496806 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496778 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.496863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.496848 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.497119 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.497097 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.498467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.498446 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.498693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.498674 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.503959 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.503940 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.504465 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.504444 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkblf\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-kube-api-access-rkblf\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-hprq7\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.629373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.629332 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:00.919967 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.919941 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7"] Nov 03 10:47:00.920644 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:47:00.920621 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56664a6c_c1b4_4e9d_a3de_0b50d2456fa6.slice/crio-30d4b0045a1680b60518731ec672579ef131ed6d9f36e74b137da33b21680776 WatchSource:0}: Error finding container 30d4b0045a1680b60518731ec672579ef131ed6d9f36e74b137da33b21680776: Status 404 returned error can't find the container with id 30d4b0045a1680b60518731ec672579ef131ed6d9f36e74b137da33b21680776 Nov 03 10:47:00.923726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.923691 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:47:00.923810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.923757 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:47:00.923810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:00.923787 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:47:01.307742 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:01.307634 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" event={"ID":"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6","Type":"ContainerStarted","Data":"33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994"} Nov 03 10:47:01.307742 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:01.307709 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" event={"ID":"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6","Type":"ContainerStarted","Data":"30d4b0045a1680b60518731ec672579ef131ed6d9f36e74b137da33b21680776"} Nov 03 10:47:01.328798 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:01.328735 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" podStartSLOduration=1.328721443 podStartE2EDuration="1.328721443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:01.326230237 +0000 UTC m=+1099.311101519" watchObservedRunningTime="2026-11-03 10:47:01.328721443 +0000 UTC m=+1099.313592725" Nov 03 10:47:01.640497 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:01.640449 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:02.640318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:02.639394 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 10:47:02.640318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:02.639469 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 10:47:03.631546 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:03.631488 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 10:47:03.631963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:03.631911 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 10:47:04.640317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:04.639621 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 10:47:04.640317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:04.639706 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 10:47:05.635661 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:05.635623 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 10:47:05.635963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:05.635934 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 10:47:06.641469 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:06.641431 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 10:47:06.641979 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:06.641498 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 10:47:07.634747 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:07.634715 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:07.635262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:07.635234 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:07.636117 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:07.636095 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:47:18.802451 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.802416 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f"] Nov 03 10:47:18.808426 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.808405 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:18.813757 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.813733 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-642fe5\"" Nov 03 10:47:18.816008 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.815983 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-642fe5-dockercfg-wtfpb\"" Nov 03 10:47:18.819010 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.818983 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f"] Nov 03 10:47:18.872544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.872510 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6"] Nov 03 10:47:18.920401 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.920363 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26cbd8b-dd63-430d-93ce-10bb4a252a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f\" (UID: \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:18.920858 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.920828 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5cfk\" (UniqueName: \"kubernetes.io/projected/d26cbd8b-dd63-430d-93ce-10bb4a252a19-kube-api-access-s5cfk\") pod \"payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f\" (UID: \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:18.922573 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.922537 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6"] Nov 03 10:47:18.922693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:18.922668 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:19.046884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.032323 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5cfk\" (UniqueName: \"kubernetes.io/projected/d26cbd8b-dd63-430d-93ce-10bb4a252a19-kube-api-access-s5cfk\") pod \"payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f\" (UID: \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:19.046884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.032535 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26cbd8b-dd63-430d-93ce-10bb4a252a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f\" (UID: \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:19.046884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.032575 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvrg8\" (UniqueName: \"kubernetes.io/projected/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-kube-api-access-qvrg8\") pod \"payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6\" (UID: \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:19.046884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.032628 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-plugins-config-volume\") pod \"payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6\" (UID: \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:19.046884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.033886 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26cbd8b-dd63-430d-93ce-10bb4a252a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f\" (UID: \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:19.078380 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.074984 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5cfk\" (UniqueName: \"kubernetes.io/projected/d26cbd8b-dd63-430d-93ce-10bb4a252a19-kube-api-access-s5cfk\") pod \"payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f\" (UID: \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:19.126715 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.126678 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:19.134118 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.134079 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrg8\" (UniqueName: \"kubernetes.io/projected/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-kube-api-access-qvrg8\") pod \"payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6\" (UID: \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:19.134263 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.134161 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-plugins-config-volume\") pod \"payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6\" (UID: \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:19.135103 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.135078 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-plugins-config-volume\") pod \"payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6\" (UID: \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:19.142262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.142233 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrg8\" (UniqueName: \"kubernetes.io/projected/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-kube-api-access-qvrg8\") pod \"payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6\" (UID: \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:19.393457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.393372 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:19.424948 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.424899 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f"] Nov 03 10:47:19.427085 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:47:19.427055 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26cbd8b_dd63_430d_93ce_10bb4a252a19.slice/crio-2495ca242b4f98e394ac8b4e405727ab8ddedbc280b0231c465d8f59e158e5da WatchSource:0}: Error finding container 2495ca242b4f98e394ac8b4e405727ab8ddedbc280b0231c465d8f59e158e5da: Status 404 returned error can't find the container with id 2495ca242b4f98e394ac8b4e405727ab8ddedbc280b0231c465d8f59e158e5da Nov 03 10:47:19.484705 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.484674 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" event={"ID":"d26cbd8b-dd63-430d-93ce-10bb4a252a19","Type":"ContainerStarted","Data":"2495ca242b4f98e394ac8b4e405727ab8ddedbc280b0231c465d8f59e158e5da"} Nov 03 10:47:19.718984 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:19.718955 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6"] Nov 03 10:47:19.719526 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:47:19.719501 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde8d95b_7cdf_4686_80e9_3aeddfcbfb06.slice/crio-5c376749bd84895a3ca5782286004576b760b99877e2fea8b7d3ba1393a2a9b4 WatchSource:0}: Error finding container 5c376749bd84895a3ca5782286004576b760b99877e2fea8b7d3ba1393a2a9b4: Status 404 returned error can't find the container with id 5c376749bd84895a3ca5782286004576b760b99877e2fea8b7d3ba1393a2a9b4 Nov 03 10:47:20.493932 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:20.493891 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" event={"ID":"d26cbd8b-dd63-430d-93ce-10bb4a252a19","Type":"ContainerStarted","Data":"2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf"} Nov 03 10:47:20.494428 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:20.493971 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:20.495790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:20.495768 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" event={"ID":"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06","Type":"ContainerStarted","Data":"0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521"} Nov 03 10:47:20.495879 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:20.495805 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" event={"ID":"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06","Type":"ContainerStarted","Data":"5c376749bd84895a3ca5782286004576b760b99877e2fea8b7d3ba1393a2a9b4"} Nov 03 10:47:20.495879 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:20.495833 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:20.514681 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:20.514637 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" podStartSLOduration=2.514624564 podStartE2EDuration="2.514624564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:20.510853086 +0000 UTC m=+1118.495724370" watchObservedRunningTime="2026-11-03 10:47:20.514624564 +0000 UTC m=+1118.499495847" Nov 03 10:47:20.526834 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:20.526794 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" podStartSLOduration=2.526786102 podStartE2EDuration="2.526786102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:20.525422363 +0000 UTC m=+1118.510293672" watchObservedRunningTime="2026-11-03 10:47:20.526786102 +0000 UTC m=+1118.511657424" Nov 03 10:47:22.293242 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.293203 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2"] Nov 03 10:47:22.293911 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.293482 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" podUID="837be16e-5f29-45fb-91c3-651647739962" containerName="maas-api" containerID="cri-o://c51486ba5dc06926c7a01986117674c298c08ad66a0adb31481112caf72bcdf3" gracePeriod=30 Nov 03 10:47:22.666117 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.666085 5086 generic.go:356] "Generic (PLEG): container finished" podID="837be16e-5f29-45fb-91c3-651647739962" containerID="c51486ba5dc06926c7a01986117674c298c08ad66a0adb31481112caf72bcdf3" exitCode=0 Nov 03 10:47:22.666267 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.666146 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" event={"ID":"837be16e-5f29-45fb-91c3-651647739962","Type":"ContainerDied","Data":"c51486ba5dc06926c7a01986117674c298c08ad66a0adb31481112caf72bcdf3"} Nov 03 10:47:22.704207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.704138 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:47:22.723004 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.722968 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-metrics-certs\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-metrics-certs\") pod \"837be16e-5f29-45fb-91c3-651647739962\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " Nov 03 10:47:22.723185 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.723053 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837be16e-5f29-45fb-91c3-651647739962-kube-api-access-hstdw\" (UniqueName: \"kubernetes.io/projected/837be16e-5f29-45fb-91c3-651647739962-kube-api-access-hstdw\") pod \"837be16e-5f29-45fb-91c3-651647739962\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " Nov 03 10:47:22.723185 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.723080 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837be16e-5f29-45fb-91c3-651647739962-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837be16e-5f29-45fb-91c3-651647739962-openshift-service-ca\") pod \"837be16e-5f29-45fb-91c3-651647739962\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " Nov 03 10:47:22.723185 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.723171 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls\") pod \"837be16e-5f29-45fb-91c3-651647739962\" (UID: \"837be16e-5f29-45fb-91c3-651647739962\") " Nov 03 10:47:22.724172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.724149 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837be16e-5f29-45fb-91c3-651647739962-openshift-service-ca" pod "837be16e-5f29-45fb-91c3-651647739962" (UID: "837be16e-5f29-45fb-91c3-651647739962"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:47:22.727504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.727480 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-metrics-certs" pod "837be16e-5f29-45fb-91c3-651647739962" (UID: "837be16e-5f29-45fb-91c3-651647739962"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:47:22.727618 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.727505 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls" pod "837be16e-5f29-45fb-91c3-651647739962" (UID: "837be16e-5f29-45fb-91c3-651647739962"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:47:22.727748 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.727730 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837be16e-5f29-45fb-91c3-651647739962-kube-api-access-hstdw" pod "837be16e-5f29-45fb-91c3-651647739962" (UID: "837be16e-5f29-45fb-91c3-651647739962"). InnerVolumeSpecName "kube-api-access-hstdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:47:22.825018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.824983 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:22.825018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.825011 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837be16e-5f29-45fb-91c3-651647739962-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:22.825018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.825022 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hstdw\" (UniqueName: \"kubernetes.io/projected/837be16e-5f29-45fb-91c3-651647739962-kube-api-access-hstdw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:22.825262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:22.825040 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837be16e-5f29-45fb-91c3-651647739962-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:23.675164 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:23.675131 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" Nov 03 10:47:23.675587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:23.675136 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2" event={"ID":"837be16e-5f29-45fb-91c3-651647739962","Type":"ContainerDied","Data":"2b5ea4dc0181667fd84c8d632fbd6a2d5385a86a135cdee6ea1bc405b7498962"} Nov 03 10:47:23.675587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:23.675318 5086 scope.go:122] "RemoveContainer" containerID="c51486ba5dc06926c7a01986117674c298c08ad66a0adb31481112caf72bcdf3" Nov 03 10:47:23.700717 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:23.700688 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2"] Nov 03 10:47:23.704209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:23.704189 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-141053-55478bddb5-cbnb2"] Nov 03 10:47:24.172378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:24.172344 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837be16e-5f29-45fb-91c3-651647739962" path="/var/lib/kubelet/pods/837be16e-5f29-45fb-91c3-651647739962/volumes" Nov 03 10:47:24.276433 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:47:24.276402 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:47:24.333536 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:24.333504 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6"] Nov 03 10:47:26.275383 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.275352 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz"] Nov 03 10:47:26.278339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.278313 5086 cpu_manager.go:414] "containerMap: removing container" podUID="837be16e-5f29-45fb-91c3-651647739962" containerName="maas-api" Nov 03 10:47:26.278472 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.278460 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837be16e-5f29-45fb-91c3-651647739962" containerName="maas-api" Nov 03 10:47:26.289851 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.289822 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="837be16e-5f29-45fb-91c3-651647739962" containerName="maas-api" Nov 03 10:47:26.300785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.300758 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz"] Nov 03 10:47:26.301054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.301040 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.306422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.306395 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-642fe5-serving-cert\"" Nov 03 10:47:26.381473 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.381439 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-metrics-certs\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.381640 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.381509 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db645df-be5b-4543-99db-ddb4f498b141-openshift-service-ca\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.381737 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.381714 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-maas-api-tls\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.381912 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.381895 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78n4f\" (UniqueName: \"kubernetes.io/projected/1db645df-be5b-4543-99db-ddb4f498b141-kube-api-access-78n4f\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.500507 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.500464 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-metrics-certs\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.512405 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.509533 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-metrics-certs\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.512760 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.512620 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db645df-be5b-4543-99db-ddb4f498b141-openshift-service-ca\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.513237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.512895 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-maas-api-tls\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.513237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.513111 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78n4f\" (UniqueName: \"kubernetes.io/projected/1db645df-be5b-4543-99db-ddb4f498b141-kube-api-access-78n4f\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.513773 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.513731 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db645df-be5b-4543-99db-ddb4f498b141-openshift-service-ca\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.520231 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.520206 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-maas-api-tls\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.524638 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.524614 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78n4f\" (UniqueName: \"kubernetes.io/projected/1db645df-be5b-4543-99db-ddb4f498b141-kube-api-access-78n4f\") pod \"maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.623066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.623032 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:26.915741 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:26.915541 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz"] Nov 03 10:47:26.925071 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:47:26.925040 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db645df_be5b_4543_99db_ddb4f498b141.slice/crio-b107ffa3ce4531b940ab9ffdfa1a0da73275f5a8e510d230f959d678ae3399e6 WatchSource:0}: Error finding container b107ffa3ce4531b940ab9ffdfa1a0da73275f5a8e510d230f959d678ae3399e6: Status 404 returned error can't find the container with id b107ffa3ce4531b940ab9ffdfa1a0da73275f5a8e510d230f959d678ae3399e6 Nov 03 10:47:27.729400 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:27.729315 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" event={"ID":"1db645df-be5b-4543-99db-ddb4f498b141","Type":"ContainerStarted","Data":"e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3"} Nov 03 10:47:27.729400 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:27.729362 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" event={"ID":"1db645df-be5b-4543-99db-ddb4f498b141","Type":"ContainerStarted","Data":"b107ffa3ce4531b940ab9ffdfa1a0da73275f5a8e510d230f959d678ae3399e6"} Nov 03 10:47:27.729836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:27.729528 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:27.750308 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:27.750251 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" podStartSLOduration=1.325247907 podStartE2EDuration="1.750237197s" totalImagesPullingTime="424.98929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:27.746702131 +0000 UTC m=+1125.731573414" watchObservedRunningTime="2026-11-03 10:47:27.750237197 +0000 UTC m=+1125.735108481" Nov 03 10:47:29.334421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.334374 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" podUID="5036ae98-bc81-458b-a5d4-24f31c86c422" containerName="payload-processing" containerID="cri-o://2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9" gracePeriod=25 Nov 03 10:47:29.641448 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.641421 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:47:29.750341 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.750308 5086 generic.go:356] "Generic (PLEG): container finished" podID="5036ae98-bc81-458b-a5d4-24f31c86c422" containerID="2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9" exitCode=0 Nov 03 10:47:29.750538 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.750373 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" Nov 03 10:47:29.750538 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.750519 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" event={"ID":"5036ae98-bc81-458b-a5d4-24f31c86c422","Type":"ContainerDied","Data":"2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9"} Nov 03 10:47:29.750661 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.750560 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6" event={"ID":"5036ae98-bc81-458b-a5d4-24f31c86c422","Type":"ContainerDied","Data":"4516d08d78096e454e3fd7e1b8d1dfa4b4cea4b4ce7a717f693c29e6eafc7f6c"} Nov 03 10:47:29.750661 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.750582 5086 scope.go:122] "RemoveContainer" containerID="2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9" Nov 03 10:47:29.762800 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.762771 5086 scope.go:122] "RemoveContainer" containerID="2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9" Nov 03 10:47:29.763136 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:47:29.763114 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9\": container with ID starting with 2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9 not found: ID does not exist" containerID="2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9" Nov 03 10:47:29.763253 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.763149 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9"} err="failed to get container status \"2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9\": rpc error: code = NotFound desc = could not find container \"2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9\": container with ID starting with 2e373ebcff4c815976cdf82479978a1c019bdf14544e907eb57ce0a9b67a61d9 not found: ID does not exist" Nov 03 10:47:29.763253 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.763121 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5036ae98-bc81-458b-a5d4-24f31c86c422-kube-api-access-s8slb\" (UniqueName: \"kubernetes.io/projected/5036ae98-bc81-458b-a5d4-24f31c86c422-kube-api-access-s8slb\") pod \"5036ae98-bc81-458b-a5d4-24f31c86c422\" (UID: \"5036ae98-bc81-458b-a5d4-24f31c86c422\") " Nov 03 10:47:29.763253 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.763246 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5036ae98-bc81-458b-a5d4-24f31c86c422-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5036ae98-bc81-458b-a5d4-24f31c86c422-plugins-config-volume\") pod \"5036ae98-bc81-458b-a5d4-24f31c86c422\" (UID: \"5036ae98-bc81-458b-a5d4-24f31c86c422\") " Nov 03 10:47:29.763887 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.763864 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5036ae98-bc81-458b-a5d4-24f31c86c422-plugins-config-volume" pod "5036ae98-bc81-458b-a5d4-24f31c86c422" (UID: "5036ae98-bc81-458b-a5d4-24f31c86c422"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:47:29.764016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.764002 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5036ae98-bc81-458b-a5d4-24f31c86c422-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:29.765347 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.765276 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5036ae98-bc81-458b-a5d4-24f31c86c422-kube-api-access-s8slb" pod "5036ae98-bc81-458b-a5d4-24f31c86c422" (UID: "5036ae98-bc81-458b-a5d4-24f31c86c422"). InnerVolumeSpecName "kube-api-access-s8slb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:47:29.865754 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:29.865679 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8slb\" (UniqueName: \"kubernetes.io/projected/5036ae98-bc81-458b-a5d4-24f31c86c422-kube-api-access-s8slb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:30.153034 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:30.152995 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6"] Nov 03 10:47:30.158346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:30.158111 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-151744-7bd85db9f5-gqmh6"] Nov 03 10:47:30.186071 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:30.186039 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5036ae98-bc81-458b-a5d4-24f31c86c422" path="/var/lib/kubelet/pods/5036ae98-bc81-458b-a5d4-24f31c86c422/volumes" Nov 03 10:47:31.506253 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:31.506218 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:47:31.506732 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:31.506288 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:47:33.742184 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:33.742103 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:47:38.970087 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:38.970059 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q"] Nov 03 10:47:38.970641 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:38.970609 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" podUID="316eabbc-0694-4267-8894-47cd1b6fd7ea" containerName="maas-api" containerID="cri-o://64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e" gracePeriod=30 Nov 03 10:47:39.419317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.419131 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:47:39.559684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.493290 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-metrics-certs\") pod \"316eabbc-0694-4267-8894-47cd1b6fd7ea\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " Nov 03 10:47:39.569246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.559908 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-maas-api-tls\") pod \"316eabbc-0694-4267-8894-47cd1b6fd7ea\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " Nov 03 10:47:39.569246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.559990 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/316eabbc-0694-4267-8894-47cd1b6fd7ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316eabbc-0694-4267-8894-47cd1b6fd7ea-openshift-service-ca\") pod \"316eabbc-0694-4267-8894-47cd1b6fd7ea\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " Nov 03 10:47:39.569246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.560036 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316eabbc-0694-4267-8894-47cd1b6fd7ea-kube-api-access-qwmjm\" (UniqueName: \"kubernetes.io/projected/316eabbc-0694-4267-8894-47cd1b6fd7ea-kube-api-access-qwmjm\") pod \"316eabbc-0694-4267-8894-47cd1b6fd7ea\" (UID: \"316eabbc-0694-4267-8894-47cd1b6fd7ea\") " Nov 03 10:47:39.569246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.562929 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316eabbc-0694-4267-8894-47cd1b6fd7ea-openshift-service-ca" pod "316eabbc-0694-4267-8894-47cd1b6fd7ea" (UID: "316eabbc-0694-4267-8894-47cd1b6fd7ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:47:39.603377 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.601411 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-metrics-certs" pod "316eabbc-0694-4267-8894-47cd1b6fd7ea" (UID: "316eabbc-0694-4267-8894-47cd1b6fd7ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:47:39.615524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.613107 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316eabbc-0694-4267-8894-47cd1b6fd7ea-kube-api-access-qwmjm" pod "316eabbc-0694-4267-8894-47cd1b6fd7ea" (UID: "316eabbc-0694-4267-8894-47cd1b6fd7ea"). InnerVolumeSpecName "kube-api-access-qwmjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:47:39.615524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.613288 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-maas-api-tls" pod "316eabbc-0694-4267-8894-47cd1b6fd7ea" (UID: "316eabbc-0694-4267-8894-47cd1b6fd7ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:47:39.662531 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.662502 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:39.662531 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.662527 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316eabbc-0694-4267-8894-47cd1b6fd7ea-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:39.662725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.662546 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316eabbc-0694-4267-8894-47cd1b6fd7ea-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:39.662725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.662556 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwmjm\" (UniqueName: \"kubernetes.io/projected/316eabbc-0694-4267-8894-47cd1b6fd7ea-kube-api-access-qwmjm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:39.871816 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.871775 5086 generic.go:356] "Generic (PLEG): container finished" podID="316eabbc-0694-4267-8894-47cd1b6fd7ea" containerID="64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e" exitCode=0 Nov 03 10:47:39.872104 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.871885 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" event={"ID":"316eabbc-0694-4267-8894-47cd1b6fd7ea","Type":"ContainerDied","Data":"64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e"} Nov 03 10:47:39.872104 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.871975 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" event={"ID":"316eabbc-0694-4267-8894-47cd1b6fd7ea","Type":"ContainerDied","Data":"ee3e50e81e11b838cffd5555c9e25f80dc593cc238b812f73e142d0930cea81e"} Nov 03 10:47:39.872104 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.872007 5086 scope.go:122] "RemoveContainer" containerID="64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e" Nov 03 10:47:39.872334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.872177 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q" Nov 03 10:47:39.887713 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.887677 5086 scope.go:122] "RemoveContainer" containerID="64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e" Nov 03 10:47:39.888159 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:47:39.888135 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e\": container with ID starting with 64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e not found: ID does not exist" containerID="64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e" Nov 03 10:47:39.888246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.888176 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e"} err="failed to get container status \"64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e\": rpc error: code = NotFound desc = could not find container \"64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e\": container with ID starting with 64b624232e7d4bb9328e5d5ebc7a2f559a4464988f05d4959c5ff668be7a734e not found: ID does not exist" Nov 03 10:47:39.902890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.902857 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q"] Nov 03 10:47:39.906629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:39.906601 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-151744-86b5b7d4cf-74s7q"] Nov 03 10:47:40.172993 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:40.172903 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316eabbc-0694-4267-8894-47cd1b6fd7ea" path="/var/lib/kubelet/pods/316eabbc-0694-4267-8894-47cd1b6fd7ea/volumes" Nov 03 10:47:44.536303 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:44.536268 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h"] Nov 03 10:47:44.549474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:44.549438 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97"] Nov 03 10:47:49.537064 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:49.537019 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" podUID="01b0e25b-1d36-4315-8a1a-d6df9df66a11" containerName="payload-processing" containerID="cri-o://37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0" gracePeriod=25 Nov 03 10:47:49.550577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:49.550546 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" podUID="a5754800-e23d-42ab-b135-12ff97a2eadf" containerName="payload-pre-processing" containerID="cri-o://81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17" gracePeriod=25 Nov 03 10:47:49.858903 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:49.858863 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql"] Nov 03 10:47:49.859207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:49.859172 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" podUID="dd442a16-582f-4de7-b474-fee20cdc185f" containerName="maas-api" containerID="cri-o://edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0" gracePeriod=30 Nov 03 10:47:50.257340 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.257290 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:47:50.359253 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.359223 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-maas-api-tls\") pod \"dd442a16-582f-4de7-b474-fee20cdc185f\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " Nov 03 10:47:50.359487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.359271 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd442a16-582f-4de7-b474-fee20cdc185f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd442a16-582f-4de7-b474-fee20cdc185f-openshift-service-ca\") pod \"dd442a16-582f-4de7-b474-fee20cdc185f\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " Nov 03 10:47:50.359487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.359383 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd442a16-582f-4de7-b474-fee20cdc185f-kube-api-access-9nzbt\" (UniqueName: \"kubernetes.io/projected/dd442a16-582f-4de7-b474-fee20cdc185f-kube-api-access-9nzbt\") pod \"dd442a16-582f-4de7-b474-fee20cdc185f\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " Nov 03 10:47:50.359487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.359456 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-metrics-certs\") pod \"dd442a16-582f-4de7-b474-fee20cdc185f\" (UID: \"dd442a16-582f-4de7-b474-fee20cdc185f\") " Nov 03 10:47:50.359758 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.359711 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd442a16-582f-4de7-b474-fee20cdc185f-openshift-service-ca" pod "dd442a16-582f-4de7-b474-fee20cdc185f" (UID: "dd442a16-582f-4de7-b474-fee20cdc185f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:47:50.360144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.360120 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd442a16-582f-4de7-b474-fee20cdc185f-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:50.362108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.362084 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-metrics-certs" pod "dd442a16-582f-4de7-b474-fee20cdc185f" (UID: "dd442a16-582f-4de7-b474-fee20cdc185f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:47:50.362268 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.362242 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-maas-api-tls" pod "dd442a16-582f-4de7-b474-fee20cdc185f" (UID: "dd442a16-582f-4de7-b474-fee20cdc185f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:47:50.363479 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.363451 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd442a16-582f-4de7-b474-fee20cdc185f-kube-api-access-9nzbt" pod "dd442a16-582f-4de7-b474-fee20cdc185f" (UID: "dd442a16-582f-4de7-b474-fee20cdc185f"). InnerVolumeSpecName "kube-api-access-9nzbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:47:50.461685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.461600 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:50.461685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.461629 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd442a16-582f-4de7-b474-fee20cdc185f-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:50.461685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.461640 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nzbt\" (UniqueName: \"kubernetes.io/projected/dd442a16-582f-4de7-b474-fee20cdc185f-kube-api-access-9nzbt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:47:50.972031 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.971998 5086 generic.go:356] "Generic (PLEG): container finished" podID="dd442a16-582f-4de7-b474-fee20cdc185f" containerID="edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0" exitCode=0 Nov 03 10:47:50.972487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.972051 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" event={"ID":"dd442a16-582f-4de7-b474-fee20cdc185f","Type":"ContainerDied","Data":"edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0"} Nov 03 10:47:50.972487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.972054 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" Nov 03 10:47:50.972487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.972100 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql" event={"ID":"dd442a16-582f-4de7-b474-fee20cdc185f","Type":"ContainerDied","Data":"4b35af4b6a43eb53a64fddd63f49fcab9f4b4e32a21fc09a003d3deb7905ddaf"} Nov 03 10:47:50.972487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.972122 5086 scope.go:122] "RemoveContainer" containerID="edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0" Nov 03 10:47:50.983808 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.983789 5086 scope.go:122] "RemoveContainer" containerID="668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e" Nov 03 10:47:50.995004 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.993984 5086 scope.go:122] "RemoveContainer" containerID="edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0" Nov 03 10:47:50.995004 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:47:50.994881 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0\": container with ID starting with edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0 not found: ID does not exist" containerID="edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0" Nov 03 10:47:50.995004 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.994919 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0"} err="failed to get container status \"edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0\": rpc error: code = NotFound desc = could not find container \"edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0\": container with ID starting with edd11bd807c8f3bbdac15ba5a59587b1f5cfc7c72ffdee929e4adbf18c90eee0 not found: ID does not exist" Nov 03 10:47:50.995004 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.994938 5086 scope.go:122] "RemoveContainer" containerID="668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e" Nov 03 10:47:50.995271 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:47:50.995201 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e\": container with ID starting with 668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e not found: ID does not exist" containerID="668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e" Nov 03 10:47:50.995271 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.995231 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e"} err="failed to get container status \"668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e\": rpc error: code = NotFound desc = could not find container \"668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e\": container with ID starting with 668276039aea4263b370ee36dc217be3cfb197bf39c8c7fc21ab6acd35e2133e not found: ID does not exist" Nov 03 10:47:50.995513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.995496 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql"] Nov 03 10:47:50.999441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:50.999422 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a917b1-864b97db97-7x8ql"] Nov 03 10:47:52.172495 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:52.172456 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd442a16-582f-4de7-b474-fee20cdc185f" path="/var/lib/kubelet/pods/dd442a16-582f-4de7-b474-fee20cdc185f/volumes" Nov 03 10:47:54.275525 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:47:54.275489 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:47:57.430342 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:57.430264 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx"] Nov 03 10:47:57.463342 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:57.463264 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr"] Nov 03 10:47:57.870363 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:57.862430 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" podUID="a5754800-e23d-42ab-b135-12ff97a2eadf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:47:57.872435 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:57.872393 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" podUID="01b0e25b-1d36-4315-8a1a-d6df9df66a11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:47:58.550350 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:58.550320 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk"] Nov 03 10:47:58.561426 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:47:58.561395 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t"] Nov 03 10:48:02.431731 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.431626 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" podUID="90f2dead-f4bf-4c3b-b1c8-0245dade0fde" containerName="payload-processing" containerID="cri-o://9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4" gracePeriod=25 Nov 03 10:48:02.466318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.465535 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" podUID="eb792017-70d8-4fc3-95e1-1cea7ef75cd0" containerName="payload-pre-processing" containerID="cri-o://f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171" gracePeriod=25 Nov 03 10:48:02.813598 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.813570 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:48:02.895748 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.895707 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-kube-api-access-t2d5j\" (UniqueName: \"kubernetes.io/projected/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-kube-api-access-t2d5j\") pod \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\" (UID: \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\") " Nov 03 10:48:02.895940 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.895774 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-plugins-config-volume\") pod \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\" (UID: \"90f2dead-f4bf-4c3b-b1c8-0245dade0fde\") " Nov 03 10:48:02.896711 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.896673 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-plugins-config-volume" pod "90f2dead-f4bf-4c3b-b1c8-0245dade0fde" (UID: "90f2dead-f4bf-4c3b-b1c8-0245dade0fde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:02.897430 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.897403 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:48:02.897535 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.897405 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:02.900195 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.900173 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-kube-api-access-t2d5j" pod "90f2dead-f4bf-4c3b-b1c8-0245dade0fde" (UID: "90f2dead-f4bf-4c3b-b1c8-0245dade0fde"). InnerVolumeSpecName "kube-api-access-t2d5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:02.998243 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.998139 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-kube-api-access-5fccf\" (UniqueName: \"kubernetes.io/projected/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-kube-api-access-5fccf\") pod \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\" (UID: \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\") " Nov 03 10:48:02.998450 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.998261 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-plugins-config-volume\") pod \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\" (UID: \"eb792017-70d8-4fc3-95e1-1cea7ef75cd0\") " Nov 03 10:48:02.998719 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.998683 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-plugins-config-volume" pod "eb792017-70d8-4fc3-95e1-1cea7ef75cd0" (UID: "eb792017-70d8-4fc3-95e1-1cea7ef75cd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:02.998952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.998935 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:02.999006 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:02.998958 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2d5j\" (UniqueName: \"kubernetes.io/projected/90f2dead-f4bf-4c3b-b1c8-0245dade0fde-kube-api-access-t2d5j\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:03.000801 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.000777 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-kube-api-access-5fccf" pod "eb792017-70d8-4fc3-95e1-1cea7ef75cd0" (UID: "eb792017-70d8-4fc3-95e1-1cea7ef75cd0"). InnerVolumeSpecName "kube-api-access-5fccf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:03.100912 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.100874 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fccf\" (UniqueName: \"kubernetes.io/projected/eb792017-70d8-4fc3-95e1-1cea7ef75cd0-kube-api-access-5fccf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:03.114215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.114180 5086 generic.go:356] "Generic (PLEG): container finished" podID="90f2dead-f4bf-4c3b-b1c8-0245dade0fde" containerID="9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4" exitCode=0 Nov 03 10:48:03.114425 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.114234 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" Nov 03 10:48:03.114425 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.114272 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" event={"ID":"90f2dead-f4bf-4c3b-b1c8-0245dade0fde","Type":"ContainerDied","Data":"9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4"} Nov 03 10:48:03.114425 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.114360 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx" event={"ID":"90f2dead-f4bf-4c3b-b1c8-0245dade0fde","Type":"ContainerDied","Data":"31699a046eebd4ec77fe1d8f964b0ed531368d1f098382ba2cd29275731c3e82"} Nov 03 10:48:03.114425 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.114389 5086 scope.go:122] "RemoveContainer" containerID="9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4" Nov 03 10:48:03.115726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.115707 5086 generic.go:356] "Generic (PLEG): container finished" podID="eb792017-70d8-4fc3-95e1-1cea7ef75cd0" containerID="f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171" exitCode=0 Nov 03 10:48:03.115836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.115751 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" Nov 03 10:48:03.115836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.115759 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" event={"ID":"eb792017-70d8-4fc3-95e1-1cea7ef75cd0","Type":"ContainerDied","Data":"f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171"} Nov 03 10:48:03.115836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.115795 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr" event={"ID":"eb792017-70d8-4fc3-95e1-1cea7ef75cd0","Type":"ContainerDied","Data":"94cbb6f881d32c5017398a32f36eb684c5dbc1c1eb18fe0e73776823ea8ffd7e"} Nov 03 10:48:03.129093 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.129071 5086 scope.go:122] "RemoveContainer" containerID="9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4" Nov 03 10:48:03.129641 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:03.129606 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4\": container with ID starting with 9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4 not found: ID does not exist" containerID="9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4" Nov 03 10:48:03.129725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.129640 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4"} err="failed to get container status \"9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4\": rpc error: code = NotFound desc = could not find container \"9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4\": container with ID starting with 9e6f3ffc3145104ba39858f2e3b81a7411b9396a86898597d611791392423bb4 not found: ID does not exist" Nov 03 10:48:03.129725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.129664 5086 scope.go:122] "RemoveContainer" containerID="f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171" Nov 03 10:48:03.139901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.139884 5086 scope.go:122] "RemoveContainer" containerID="f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171" Nov 03 10:48:03.140154 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:03.140127 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171\": container with ID starting with f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171 not found: ID does not exist" containerID="f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171" Nov 03 10:48:03.140251 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.140164 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171"} err="failed to get container status \"f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171\": rpc error: code = NotFound desc = could not find container \"f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171\": container with ID starting with f0f6f2a5543428f1679e39fb7da565f6296ff055f4f40cc0c8ff669d97083171 not found: ID does not exist" Nov 03 10:48:03.145488 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.145467 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr"] Nov 03 10:48:03.152901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.152882 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-273012-59dcd47f9f-28whr"] Nov 03 10:48:03.162686 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.162660 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx"] Nov 03 10:48:03.165869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.165850 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-273012-8665bbb447-4h8kx"] Nov 03 10:48:03.550631 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.550593 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" podUID="50243e18-189d-4f8d-a3a2-f382b90af7d6" containerName="payload-processing" containerID="cri-o://de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c" gracePeriod=25 Nov 03 10:48:03.563702 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.563653 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" podUID="c5b25364-0fa4-4687-9c08-d48e2f9eba7d" containerName="payload-pre-processing" containerID="cri-o://58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c" gracePeriod=25 Nov 03 10:48:03.897550 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.897527 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:48:03.953369 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:03.953336 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:48:04.035430 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.035344 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-kube-api-access-l5r94\" (UniqueName: \"kubernetes.io/projected/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-kube-api-access-l5r94\") pod \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\" (UID: \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\") " Nov 03 10:48:04.035430 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.035425 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50243e18-189d-4f8d-a3a2-f382b90af7d6-kube-api-access-vhvjn\" (UniqueName: \"kubernetes.io/projected/50243e18-189d-4f8d-a3a2-f382b90af7d6-kube-api-access-vhvjn\") pod \"50243e18-189d-4f8d-a3a2-f382b90af7d6\" (UID: \"50243e18-189d-4f8d-a3a2-f382b90af7d6\") " Nov 03 10:48:04.035678 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.035546 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50243e18-189d-4f8d-a3a2-f382b90af7d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50243e18-189d-4f8d-a3a2-f382b90af7d6-plugins-config-volume\") pod \"50243e18-189d-4f8d-a3a2-f382b90af7d6\" (UID: \"50243e18-189d-4f8d-a3a2-f382b90af7d6\") " Nov 03 10:48:04.035744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.035707 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-plugins-config-volume\") pod \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\" (UID: \"c5b25364-0fa4-4687-9c08-d48e2f9eba7d\") " Nov 03 10:48:04.036251 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.036220 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50243e18-189d-4f8d-a3a2-f382b90af7d6-plugins-config-volume" pod "50243e18-189d-4f8d-a3a2-f382b90af7d6" (UID: "50243e18-189d-4f8d-a3a2-f382b90af7d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:04.036540 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.036385 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50243e18-189d-4f8d-a3a2-f382b90af7d6-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:04.036640 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.036548 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-plugins-config-volume" pod "c5b25364-0fa4-4687-9c08-d48e2f9eba7d" (UID: "c5b25364-0fa4-4687-9c08-d48e2f9eba7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:04.037913 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.037895 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-kube-api-access-l5r94" pod "c5b25364-0fa4-4687-9c08-d48e2f9eba7d" (UID: "c5b25364-0fa4-4687-9c08-d48e2f9eba7d"). InnerVolumeSpecName "kube-api-access-l5r94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:04.038280 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.038264 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50243e18-189d-4f8d-a3a2-f382b90af7d6-kube-api-access-vhvjn" pod "50243e18-189d-4f8d-a3a2-f382b90af7d6" (UID: "50243e18-189d-4f8d-a3a2-f382b90af7d6"). InnerVolumeSpecName "kube-api-access-vhvjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:04.127615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.127577 5086 generic.go:356] "Generic (PLEG): container finished" podID="50243e18-189d-4f8d-a3a2-f382b90af7d6" containerID="de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c" exitCode=0 Nov 03 10:48:04.127909 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.127627 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" Nov 03 10:48:04.127909 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.127657 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" event={"ID":"50243e18-189d-4f8d-a3a2-f382b90af7d6","Type":"ContainerDied","Data":"de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c"} Nov 03 10:48:04.127909 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.127728 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk" event={"ID":"50243e18-189d-4f8d-a3a2-f382b90af7d6","Type":"ContainerDied","Data":"28bc3038b3b2b4481e889c93ca85b842031a685ef71b38b417871130967fe851"} Nov 03 10:48:04.127909 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.127758 5086 scope.go:122] "RemoveContainer" containerID="de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c" Nov 03 10:48:04.129682 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.129645 5086 generic.go:356] "Generic (PLEG): container finished" podID="c5b25364-0fa4-4687-9c08-d48e2f9eba7d" containerID="58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c" exitCode=0 Nov 03 10:48:04.129814 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.129769 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" event={"ID":"c5b25364-0fa4-4687-9c08-d48e2f9eba7d","Type":"ContainerDied","Data":"58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c"} Nov 03 10:48:04.129814 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.129808 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" event={"ID":"c5b25364-0fa4-4687-9c08-d48e2f9eba7d","Type":"ContainerDied","Data":"30a66c7457d0c923d789c395a36f579aee46d220f6f9c5850c7a9fdfd8f3f9f6"} Nov 03 10:48:04.129942 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.129880 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t" Nov 03 10:48:04.138275 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.138248 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:04.138550 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.138524 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5r94\" (UniqueName: \"kubernetes.io/projected/c5b25364-0fa4-4687-9c08-d48e2f9eba7d-kube-api-access-l5r94\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:04.138668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.138563 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhvjn\" (UniqueName: \"kubernetes.io/projected/50243e18-189d-4f8d-a3a2-f382b90af7d6-kube-api-access-vhvjn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:04.147016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.146983 5086 scope.go:122] "RemoveContainer" containerID="de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c" Nov 03 10:48:04.147327 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:04.147290 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c\": container with ID starting with de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c not found: ID does not exist" containerID="de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c" Nov 03 10:48:04.147413 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.147335 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c"} err="failed to get container status \"de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c\": rpc error: code = NotFound desc = could not find container \"de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c\": container with ID starting with de134672fc139d1d23ede4c52ca4bad82c7425b14d78cea3edab6c1758f67f5c not found: ID does not exist" Nov 03 10:48:04.147413 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.147354 5086 scope.go:122] "RemoveContainer" containerID="58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c" Nov 03 10:48:04.158169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.158147 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk"] Nov 03 10:48:04.158792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.158776 5086 scope.go:122] "RemoveContainer" containerID="58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c" Nov 03 10:48:04.159055 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:04.159037 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c\": container with ID starting with 58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c not found: ID does not exist" containerID="58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c" Nov 03 10:48:04.159138 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.159061 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c"} err="failed to get container status \"58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c\": rpc error: code = NotFound desc = could not find container \"58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c\": container with ID starting with 58ab054d6a3e6eea99c4b7b49e757657fe98ca39060c386be205f575d570771c not found: ID does not exist" Nov 03 10:48:04.161949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.161925 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d8392-fd6bdf685-cxsnk"] Nov 03 10:48:04.175275 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.175247 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50243e18-189d-4f8d-a3a2-f382b90af7d6" path="/var/lib/kubelet/pods/50243e18-189d-4f8d-a3a2-f382b90af7d6/volumes" Nov 03 10:48:04.175794 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.175779 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f2dead-f4bf-4c3b-b1c8-0245dade0fde" path="/var/lib/kubelet/pods/90f2dead-f4bf-4c3b-b1c8-0245dade0fde/volumes" Nov 03 10:48:04.176170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.176157 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb792017-70d8-4fc3-95e1-1cea7ef75cd0" path="/var/lib/kubelet/pods/eb792017-70d8-4fc3-95e1-1cea7ef75cd0/volumes" Nov 03 10:48:04.176494 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.176480 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t"] Nov 03 10:48:04.176534 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:04.176512 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d8392-c5d8585cf-kd56t"] Nov 03 10:48:06.172129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:06.172098 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b25364-0fa4-4687-9c08-d48e2f9eba7d" path="/var/lib/kubelet/pods/c5b25364-0fa4-4687-9c08-d48e2f9eba7d/volumes" Nov 03 10:48:06.950494 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:06.950462 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd"] Nov 03 10:48:06.950744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:06.950711 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" podUID="3171b0ed-e03b-4466-a398-9568f50cb92b" containerName="maas-api" containerID="cri-o://39ae33e7ce184812b6571c6c776c12796d8a9f5098cb3724b45dcddeb8234d5a" gracePeriod=30 Nov 03 10:48:07.366913 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.366659 5086 generic.go:356] "Generic (PLEG): container finished" podID="3171b0ed-e03b-4466-a398-9568f50cb92b" containerID="39ae33e7ce184812b6571c6c776c12796d8a9f5098cb3724b45dcddeb8234d5a" exitCode=0 Nov 03 10:48:07.368328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.367940 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" event={"ID":"3171b0ed-e03b-4466-a398-9568f50cb92b","Type":"ContainerDied","Data":"39ae33e7ce184812b6571c6c776c12796d8a9f5098cb3724b45dcddeb8234d5a"} Nov 03 10:48:07.414952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.414925 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:48:07.550327 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.545400 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3171b0ed-e03b-4466-a398-9568f50cb92b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3171b0ed-e03b-4466-a398-9568f50cb92b-openshift-service-ca\") pod \"3171b0ed-e03b-4466-a398-9568f50cb92b\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " Nov 03 10:48:07.550327 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.545476 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-metrics-certs\") pod \"3171b0ed-e03b-4466-a398-9568f50cb92b\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " Nov 03 10:48:07.550327 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.545541 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3171b0ed-e03b-4466-a398-9568f50cb92b-kube-api-access-zk8fz\" (UniqueName: \"kubernetes.io/projected/3171b0ed-e03b-4466-a398-9568f50cb92b-kube-api-access-zk8fz\") pod \"3171b0ed-e03b-4466-a398-9568f50cb92b\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " Nov 03 10:48:07.550327 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.545614 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-maas-api-tls\") pod \"3171b0ed-e03b-4466-a398-9568f50cb92b\" (UID: \"3171b0ed-e03b-4466-a398-9568f50cb92b\") " Nov 03 10:48:07.552803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.551778 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3171b0ed-e03b-4466-a398-9568f50cb92b-openshift-service-ca" pod "3171b0ed-e03b-4466-a398-9568f50cb92b" (UID: "3171b0ed-e03b-4466-a398-9568f50cb92b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:07.554731 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.554693 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-metrics-certs" pod "3171b0ed-e03b-4466-a398-9568f50cb92b" (UID: "3171b0ed-e03b-4466-a398-9568f50cb92b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:48:07.555120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.555091 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3171b0ed-e03b-4466-a398-9568f50cb92b-kube-api-access-zk8fz" pod "3171b0ed-e03b-4466-a398-9568f50cb92b" (UID: "3171b0ed-e03b-4466-a398-9568f50cb92b"). InnerVolumeSpecName "kube-api-access-zk8fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:07.561750 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.561705 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-maas-api-tls" pod "3171b0ed-e03b-4466-a398-9568f50cb92b" (UID: "3171b0ed-e03b-4466-a398-9568f50cb92b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:48:07.647636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.647557 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3171b0ed-e03b-4466-a398-9568f50cb92b-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:07.647636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.647587 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:07.647636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.647597 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8fz\" (UniqueName: \"kubernetes.io/projected/3171b0ed-e03b-4466-a398-9568f50cb92b-kube-api-access-zk8fz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:07.647636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.647606 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3171b0ed-e03b-4466-a398-9568f50cb92b-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:07.828951 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.828743 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" podUID="a5754800-e23d-42ab-b135-12ff97a2eadf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:48:07.829229 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:07.828960 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" podUID="01b0e25b-1d36-4315-8a1a-d6df9df66a11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:48:08.378803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:08.378775 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" Nov 03 10:48:08.378803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:08.378791 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd" event={"ID":"3171b0ed-e03b-4466-a398-9568f50cb92b","Type":"ContainerDied","Data":"5a08fe8b2ed5bc839a3495c14fb960155547008db9977d016ba56cfea2f5000b"} Nov 03 10:48:08.379392 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:08.378856 5086 scope.go:122] "RemoveContainer" containerID="39ae33e7ce184812b6571c6c776c12796d8a9f5098cb3724b45dcddeb8234d5a" Nov 03 10:48:08.396832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:08.396801 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd"] Nov 03 10:48:08.401441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:08.401417 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-b88hd"] Nov 03 10:48:09.903836 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:09.903801 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:48:09.907567 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:09.907543 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:48:10.089502 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.089469 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5754800-e23d-42ab-b135-12ff97a2eadf-kube-api-access-bqsjp\" (UniqueName: \"kubernetes.io/projected/a5754800-e23d-42ab-b135-12ff97a2eadf-kube-api-access-bqsjp\") pod \"a5754800-e23d-42ab-b135-12ff97a2eadf\" (UID: \"a5754800-e23d-42ab-b135-12ff97a2eadf\") " Nov 03 10:48:10.089702 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.089515 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5754800-e23d-42ab-b135-12ff97a2eadf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5754800-e23d-42ab-b135-12ff97a2eadf-plugins-config-volume\") pod \"a5754800-e23d-42ab-b135-12ff97a2eadf\" (UID: \"a5754800-e23d-42ab-b135-12ff97a2eadf\") " Nov 03 10:48:10.089702 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.089655 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b0e25b-1d36-4315-8a1a-d6df9df66a11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b0e25b-1d36-4315-8a1a-d6df9df66a11-plugins-config-volume\") pod \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\" (UID: \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\") " Nov 03 10:48:10.089825 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.089700 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b0e25b-1d36-4315-8a1a-d6df9df66a11-kube-api-access-p2d6x\" (UniqueName: \"kubernetes.io/projected/01b0e25b-1d36-4315-8a1a-d6df9df66a11-kube-api-access-p2d6x\") pod \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\" (UID: \"01b0e25b-1d36-4315-8a1a-d6df9df66a11\") " Nov 03 10:48:10.090037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.090009 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5754800-e23d-42ab-b135-12ff97a2eadf-plugins-config-volume" pod "a5754800-e23d-42ab-b135-12ff97a2eadf" (UID: "a5754800-e23d-42ab-b135-12ff97a2eadf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:10.090144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.090043 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b0e25b-1d36-4315-8a1a-d6df9df66a11-plugins-config-volume" pod "01b0e25b-1d36-4315-8a1a-d6df9df66a11" (UID: "01b0e25b-1d36-4315-8a1a-d6df9df66a11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:10.090285 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.090268 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5754800-e23d-42ab-b135-12ff97a2eadf-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:10.090367 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.090287 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b0e25b-1d36-4315-8a1a-d6df9df66a11-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:10.091854 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.091830 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5754800-e23d-42ab-b135-12ff97a2eadf-kube-api-access-bqsjp" pod "a5754800-e23d-42ab-b135-12ff97a2eadf" (UID: "a5754800-e23d-42ab-b135-12ff97a2eadf"). InnerVolumeSpecName "kube-api-access-bqsjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:10.091956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.091900 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b0e25b-1d36-4315-8a1a-d6df9df66a11-kube-api-access-p2d6x" pod "01b0e25b-1d36-4315-8a1a-d6df9df66a11" (UID: "01b0e25b-1d36-4315-8a1a-d6df9df66a11"). InnerVolumeSpecName "kube-api-access-p2d6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:10.171961 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.171882 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3171b0ed-e03b-4466-a398-9568f50cb92b" path="/var/lib/kubelet/pods/3171b0ed-e03b-4466-a398-9568f50cb92b/volumes" Nov 03 10:48:10.191778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.191752 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqsjp\" (UniqueName: \"kubernetes.io/projected/a5754800-e23d-42ab-b135-12ff97a2eadf-kube-api-access-bqsjp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:10.191778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.191777 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2d6x\" (UniqueName: \"kubernetes.io/projected/01b0e25b-1d36-4315-8a1a-d6df9df66a11-kube-api-access-p2d6x\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:10.396378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.395926 5086 generic.go:356] "Generic (PLEG): container finished" podID="01b0e25b-1d36-4315-8a1a-d6df9df66a11" containerID="37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0" exitCode=0 Nov 03 10:48:10.396378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.396026 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" event={"ID":"01b0e25b-1d36-4315-8a1a-d6df9df66a11","Type":"ContainerDied","Data":"37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0"} Nov 03 10:48:10.396378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.396121 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" event={"ID":"01b0e25b-1d36-4315-8a1a-d6df9df66a11","Type":"ContainerDied","Data":"09382310d743b7587493ecf3800ac5fd28365cd8b72c13cfd90a16597c723e72"} Nov 03 10:48:10.396378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.396141 5086 scope.go:122] "RemoveContainer" containerID="37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0" Nov 03 10:48:10.396684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.396464 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h" Nov 03 10:48:10.398652 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.398605 5086 generic.go:356] "Generic (PLEG): container finished" podID="a5754800-e23d-42ab-b135-12ff97a2eadf" containerID="81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17" exitCode=0 Nov 03 10:48:10.398790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.398673 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" event={"ID":"a5754800-e23d-42ab-b135-12ff97a2eadf","Type":"ContainerDied","Data":"81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17"} Nov 03 10:48:10.398790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.398751 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" event={"ID":"a5754800-e23d-42ab-b135-12ff97a2eadf","Type":"ContainerDied","Data":"c98b1dbc7337c39f15c62eaa24cf4ab70d9796073fff9619395646a4f5227a4b"} Nov 03 10:48:10.398881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.398836 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97" Nov 03 10:48:10.408321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.408285 5086 scope.go:122] "RemoveContainer" containerID="37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0" Nov 03 10:48:10.408613 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:10.408592 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0\": container with ID starting with 37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0 not found: ID does not exist" containerID="37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0" Nov 03 10:48:10.408679 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.408623 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0"} err="failed to get container status \"37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0\": rpc error: code = NotFound desc = could not find container \"37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0\": container with ID starting with 37d03a8e9abe6207187c90f1b6e00c4ac56bd2049d8f0fbf75c0f17b2c3b34f0 not found: ID does not exist" Nov 03 10:48:10.408679 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.408639 5086 scope.go:122] "RemoveContainer" containerID="81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17" Nov 03 10:48:10.419347 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.419323 5086 scope.go:122] "RemoveContainer" containerID="81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17" Nov 03 10:48:10.419756 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:10.419729 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17\": container with ID starting with 81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17 not found: ID does not exist" containerID="81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17" Nov 03 10:48:10.419850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.419769 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17"} err="failed to get container status \"81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17\": rpc error: code = NotFound desc = could not find container \"81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17\": container with ID starting with 81b7a484e255d71569d39b73e0d2b9fca440ca43e932e6dbae141fc999ea6f17 not found: ID does not exist" Nov 03 10:48:10.420891 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.420866 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97"] Nov 03 10:48:10.424563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.424507 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-gpt97"] Nov 03 10:48:10.432409 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.432382 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h"] Nov 03 10:48:10.435624 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:10.435601 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-t454h"] Nov 03 10:48:11.466321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.466269 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf"] Nov 03 10:48:11.466801 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.466564 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" podUID="24126f15-928a-4852-a152-28584eca5081" containerName="maas-api" containerID="cri-o://43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1" gracePeriod=30 Nov 03 10:48:11.765109 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.765081 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:48:11.938334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.937847 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24126f15-928a-4852-a152-28584eca5081-kube-api-access-bm4gp\" (UniqueName: \"kubernetes.io/projected/24126f15-928a-4852-a152-28584eca5081-kube-api-access-bm4gp\") pod \"24126f15-928a-4852-a152-28584eca5081\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " Nov 03 10:48:11.938334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.937928 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-metrics-certs\") pod \"24126f15-928a-4852-a152-28584eca5081\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " Nov 03 10:48:11.938334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.938007 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24126f15-928a-4852-a152-28584eca5081-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24126f15-928a-4852-a152-28584eca5081-openshift-service-ca\") pod \"24126f15-928a-4852-a152-28584eca5081\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " Nov 03 10:48:11.938334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.938107 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-maas-api-tls\") pod \"24126f15-928a-4852-a152-28584eca5081\" (UID: \"24126f15-928a-4852-a152-28584eca5081\") " Nov 03 10:48:11.972321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.971494 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24126f15-928a-4852-a152-28584eca5081-openshift-service-ca" pod "24126f15-928a-4852-a152-28584eca5081" (UID: "24126f15-928a-4852-a152-28584eca5081"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:11.985173 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.981543 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24126f15-928a-4852-a152-28584eca5081-kube-api-access-bm4gp" pod "24126f15-928a-4852-a152-28584eca5081" (UID: "24126f15-928a-4852-a152-28584eca5081"). InnerVolumeSpecName "kube-api-access-bm4gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:11.985173 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.982607 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-maas-api-tls" pod "24126f15-928a-4852-a152-28584eca5081" (UID: "24126f15-928a-4852-a152-28584eca5081"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:48:11.986002 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:11.985952 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-metrics-certs" pod "24126f15-928a-4852-a152-28584eca5081" (UID: "24126f15-928a-4852-a152-28584eca5081"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:48:12.047097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.045569 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm4gp\" (UniqueName: \"kubernetes.io/projected/24126f15-928a-4852-a152-28584eca5081-kube-api-access-bm4gp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:12.047097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.045609 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:12.047097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.045625 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24126f15-928a-4852-a152-28584eca5081-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:12.047097 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.045639 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24126f15-928a-4852-a152-28584eca5081-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:12.173231 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.173194 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b0e25b-1d36-4315-8a1a-d6df9df66a11" path="/var/lib/kubelet/pods/01b0e25b-1d36-4315-8a1a-d6df9df66a11/volumes" Nov 03 10:48:12.173724 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.173709 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5754800-e23d-42ab-b135-12ff97a2eadf" path="/var/lib/kubelet/pods/a5754800-e23d-42ab-b135-12ff97a2eadf/volumes" Nov 03 10:48:12.416532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.416493 5086 generic.go:356] "Generic (PLEG): container finished" podID="24126f15-928a-4852-a152-28584eca5081" containerID="43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1" exitCode=0 Nov 03 10:48:12.416736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.416546 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" Nov 03 10:48:12.416736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.416578 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" event={"ID":"24126f15-928a-4852-a152-28584eca5081","Type":"ContainerDied","Data":"43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1"} Nov 03 10:48:12.416736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.416633 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf" event={"ID":"24126f15-928a-4852-a152-28584eca5081","Type":"ContainerDied","Data":"fd359e394a76816719f321ace976bfc006f53897e5efdc96a1ffe3243f3be0e0"} Nov 03 10:48:12.416736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.416656 5086 scope.go:122] "RemoveContainer" containerID="43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1" Nov 03 10:48:12.431684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.431664 5086 scope.go:122] "RemoveContainer" containerID="43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1" Nov 03 10:48:12.432032 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:12.432014 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1\": container with ID starting with 43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1 not found: ID does not exist" containerID="43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1" Nov 03 10:48:12.432081 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.432041 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1"} err="failed to get container status \"43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1\": rpc error: code = NotFound desc = could not find container \"43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1\": container with ID starting with 43416b823ed47b0c2cb42156a78453855e8572217c52903e9660b4c3c8d0afb1 not found: ID does not exist" Nov 03 10:48:12.445704 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.445670 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf"] Nov 03 10:48:12.452024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:12.451992 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-273012-59b7568769-nb5vf"] Nov 03 10:48:14.172334 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:14.172262 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24126f15-928a-4852-a152-28584eca5081" path="/var/lib/kubelet/pods/24126f15-928a-4852-a152-28584eca5081/volumes" Nov 03 10:48:18.749686 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.749654 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg"] Nov 03 10:48:18.750366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750350 5086 cpu_manager.go:414] "containerMap: removing container" podUID="3171b0ed-e03b-4466-a398-9568f50cb92b" containerName="maas-api" Nov 03 10:48:18.750432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750390 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3171b0ed-e03b-4466-a398-9568f50cb92b" containerName="maas-api" Nov 03 10:48:18.750432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750405 5086 cpu_manager.go:414] "containerMap: removing container" podUID="dd442a16-582f-4de7-b474-fee20cdc185f" containerName="maas-api" Nov 03 10:48:18.750432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750411 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd442a16-582f-4de7-b474-fee20cdc185f" containerName="maas-api" Nov 03 10:48:18.750432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750422 5086 cpu_manager.go:414] "containerMap: removing container" podUID="dd442a16-582f-4de7-b474-fee20cdc185f" containerName="maas-api" Nov 03 10:48:18.750432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750428 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd442a16-582f-4de7-b474-fee20cdc185f" containerName="maas-api" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750434 5086 cpu_manager.go:414] "containerMap: removing container" podUID="90f2dead-f4bf-4c3b-b1c8-0245dade0fde" containerName="payload-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750442 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f2dead-f4bf-4c3b-b1c8-0245dade0fde" containerName="payload-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750453 5086 cpu_manager.go:414] "containerMap: removing container" podUID="a5754800-e23d-42ab-b135-12ff97a2eadf" containerName="payload-pre-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750460 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5754800-e23d-42ab-b135-12ff97a2eadf" containerName="payload-pre-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750467 5086 cpu_manager.go:414] "containerMap: removing container" podUID="01b0e25b-1d36-4315-8a1a-d6df9df66a11" containerName="payload-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750472 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b0e25b-1d36-4315-8a1a-d6df9df66a11" containerName="payload-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750482 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c5b25364-0fa4-4687-9c08-d48e2f9eba7d" containerName="payload-pre-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750487 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b25364-0fa4-4687-9c08-d48e2f9eba7d" containerName="payload-pre-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750510 5086 cpu_manager.go:414] "containerMap: removing container" podUID="50243e18-189d-4f8d-a3a2-f382b90af7d6" containerName="payload-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750514 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50243e18-189d-4f8d-a3a2-f382b90af7d6" containerName="payload-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750524 5086 cpu_manager.go:414] "containerMap: removing container" podUID="eb792017-70d8-4fc3-95e1-1cea7ef75cd0" containerName="payload-pre-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750528 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb792017-70d8-4fc3-95e1-1cea7ef75cd0" containerName="payload-pre-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750537 5086 cpu_manager.go:414] "containerMap: removing container" podUID="5036ae98-bc81-458b-a5d4-24f31c86c422" containerName="payload-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750541 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5036ae98-bc81-458b-a5d4-24f31c86c422" containerName="payload-processing" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750549 5086 cpu_manager.go:414] "containerMap: removing container" podUID="24126f15-928a-4852-a152-28584eca5081" containerName="maas-api" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750553 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24126f15-928a-4852-a152-28584eca5081" containerName="maas-api" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750563 5086 cpu_manager.go:414] "containerMap: removing container" podUID="316eabbc-0694-4267-8894-47cd1b6fd7ea" containerName="maas-api" Nov 03 10:48:18.750584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750567 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316eabbc-0694-4267-8894-47cd1b6fd7ea" containerName="maas-api" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750677 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb792017-70d8-4fc3-95e1-1cea7ef75cd0" containerName="payload-pre-processing" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750685 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="5036ae98-bc81-458b-a5d4-24f31c86c422" containerName="payload-processing" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750691 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="24126f15-928a-4852-a152-28584eca5081" containerName="maas-api" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750699 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="316eabbc-0694-4267-8894-47cd1b6fd7ea" containerName="maas-api" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750709 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="3171b0ed-e03b-4466-a398-9568f50cb92b" containerName="maas-api" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750714 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd442a16-582f-4de7-b474-fee20cdc185f" containerName="maas-api" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750720 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd442a16-582f-4de7-b474-fee20cdc185f" containerName="maas-api" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750726 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f2dead-f4bf-4c3b-b1c8-0245dade0fde" containerName="payload-processing" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750731 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5754800-e23d-42ab-b135-12ff97a2eadf" containerName="payload-pre-processing" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750735 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b0e25b-1d36-4315-8a1a-d6df9df66a11" containerName="payload-processing" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750758 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b25364-0fa4-4687-9c08-d48e2f9eba7d" containerName="payload-pre-processing" Nov 03 10:48:18.751088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.750766 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="50243e18-189d-4f8d-a3a2-f382b90af7d6" containerName="payload-processing" Nov 03 10:48:18.756045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.756022 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:18.761267 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.761236 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg"] Nov 03 10:48:18.761456 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.761397 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ab592834\"" Nov 03 10:48:18.762364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.762339 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ab592834-dockercfg-n2d6n\"" Nov 03 10:48:18.766698 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.766669 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn"] Nov 03 10:48:18.771805 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.771783 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:18.779795 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.779759 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn"] Nov 03 10:48:18.896324 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.895952 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-kube-api-access-ccff6\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg\" (UID: \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:18.896324 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.896067 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn\" (UID: \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:18.896324 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.896124 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6hf\" (UniqueName: \"kubernetes.io/projected/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-kube-api-access-7g6hf\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn\" (UID: \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:18.896324 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.896201 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-plugins-config-volume\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg\" (UID: \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:18.998394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.998354 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-kube-api-access-ccff6\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg\" (UID: \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:18.998577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.998456 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn\" (UID: \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:18.998577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.998516 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6hf\" (UniqueName: \"kubernetes.io/projected/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-kube-api-access-7g6hf\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn\" (UID: \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:18.998704 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:18.998602 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-plugins-config-volume\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg\" (UID: \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:19.002835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.002751 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn\" (UID: \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:19.003956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.003873 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-plugins-config-volume\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg\" (UID: \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:19.072742 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.072651 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-kube-api-access-ccff6\") pod \"payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg\" (UID: \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\") " pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:19.076037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.076005 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6hf\" (UniqueName: \"kubernetes.io/projected/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-kube-api-access-7g6hf\") pod \"payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn\" (UID: \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:19.079741 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.079525 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:19.103238 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.103198 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:19.422040 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.422000 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg"] Nov 03 10:48:19.422447 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:48:19.422420 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe0f981_a314_4e19_9d4b_cce9b3c08e6e.slice/crio-1dd55cd028d8f71e21462a8a9060f5b3a9071383a9a7ca5fc0e13f9fe1c15822 WatchSource:0}: Error finding container 1dd55cd028d8f71e21462a8a9060f5b3a9071383a9a7ca5fc0e13f9fe1c15822: Status 404 returned error can't find the container with id 1dd55cd028d8f71e21462a8a9060f5b3a9071383a9a7ca5fc0e13f9fe1c15822 Nov 03 10:48:19.491036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.490984 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn"] Nov 03 10:48:19.509247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.509169 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" event={"ID":"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e","Type":"ContainerStarted","Data":"1dd55cd028d8f71e21462a8a9060f5b3a9071383a9a7ca5fc0e13f9fe1c15822"} Nov 03 10:48:19.511467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:19.511431 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" event={"ID":"7e5648eb-b433-40b1-bc9b-b1d1f7106d81","Type":"ContainerStarted","Data":"3417c7b4c3bd3b538c4cfd649c4c5695530eaa133835b289f732613e7923f226"} Nov 03 10:48:20.591323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:20.590386 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" event={"ID":"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e","Type":"ContainerStarted","Data":"2a575e6ad82300753acce0554eab809d6a0cf9dd2532056a6ad4f7e48383d545"} Nov 03 10:48:20.591960 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:20.591356 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:20.594001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:20.593970 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" event={"ID":"7e5648eb-b433-40b1-bc9b-b1d1f7106d81","Type":"ContainerStarted","Data":"a84e1d47c3be80eaccd57b3571a7d6bf80a7945ef4aa9443ada5df45a60bc0bd"} Nov 03 10:48:20.594641 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:20.594620 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:20.615255 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:20.615191 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" podStartSLOduration=2.615170401 podStartE2EDuration="2.615170401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:20.605143651 +0000 UTC m=+1178.590014940" watchObservedRunningTime="2026-11-03 10:48:20.615170401 +0000 UTC m=+1178.600041685" Nov 03 10:48:20.630826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:20.630615 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" podStartSLOduration=2.63059498 podStartE2EDuration="2.63059498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:20.629324234 +0000 UTC m=+1178.614195517" watchObservedRunningTime="2026-11-03 10:48:20.63059498 +0000 UTC m=+1178.615466284" Nov 03 10:48:24.041142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.041102 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml"] Nov 03 10:48:24.059066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.047075 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.059066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.051120 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ab592834-serving-cert\"" Nov 03 10:48:24.059066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.055491 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml"] Nov 03 10:48:24.270890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.270839 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-metrics-certs\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.271087 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.270902 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmfj\" (UniqueName: \"kubernetes.io/projected/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-kube-api-access-zxmfj\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.271087 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.270964 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-openshift-service-ca\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.271312 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.271280 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-maas-api-tls\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.333221 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:24.333171 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:48:24.372970 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.372937 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-maas-api-tls\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.373170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.373084 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-metrics-certs\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.373170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.373126 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmfj\" (UniqueName: \"kubernetes.io/projected/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-kube-api-access-zxmfj\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.373310 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.373175 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-openshift-service-ca\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.374287 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.374260 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-openshift-service-ca\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.376553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.376532 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-maas-api-tls\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.376860 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.376838 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-metrics-certs\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.382340 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.382313 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmfj\" (UniqueName: \"kubernetes.io/projected/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-kube-api-access-zxmfj\") pod \"maas-api-e2e-mt-ab592834-8448b97db-h2fml\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.388548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.388524 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:24.693526 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:24.693406 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml"] Nov 03 10:48:25.632231 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.632194 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" event={"ID":"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d","Type":"ContainerStarted","Data":"92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f"} Nov 03 10:48:25.632683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.632263 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" event={"ID":"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d","Type":"ContainerStarted","Data":"06212282dceab04266f047c9b1d41fbf7c6622e58b6c549a941c2485bc5a57e2"} Nov 03 10:48:25.632683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.632457 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:25.641653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.641624 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k"] Nov 03 10:48:25.646289 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.646268 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:25.652270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.652242 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k"] Nov 03 10:48:25.654642 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.654622 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-90bad4-dockercfg-c9pdj\"" Nov 03 10:48:25.654642 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.654622 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-90bad4\"" Nov 03 10:48:25.656785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.656718 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" podStartSLOduration=1.059033913 podStartE2EDuration="1.656699754s" totalImagesPullingTime="597.665841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:25.652779871 +0000 UTC m=+1183.637651155" watchObservedRunningTime="2026-11-03 10:48:25.656699754 +0000 UTC m=+1183.641571040" Nov 03 10:48:25.693859 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.693810 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582"] Nov 03 10:48:25.698847 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.698816 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:25.701577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.701133 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx6fd\" (UniqueName: \"kubernetes.io/projected/c404f7f7-80ab-418e-acae-f76716ca7b16-kube-api-access-tx6fd\") pod \"payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k\" (UID: \"c404f7f7-80ab-418e-acae-f76716ca7b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:25.701577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.701269 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c404f7f7-80ab-418e-acae-f76716ca7b16-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k\" (UID: \"c404f7f7-80ab-418e-acae-f76716ca7b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:25.720239 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.720204 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582"] Nov 03 10:48:25.810384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.804217 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq7lq\" (UniqueName: \"kubernetes.io/projected/c3a0714d-13e6-4863-a6f2-79e7e883e73f-kube-api-access-zq7lq\") pod \"payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582\" (UID: \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\") " pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:25.810384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.804308 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c404f7f7-80ab-418e-acae-f76716ca7b16-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k\" (UID: \"c404f7f7-80ab-418e-acae-f76716ca7b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:25.810384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.804524 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a0714d-13e6-4863-a6f2-79e7e883e73f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582\" (UID: \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\") " pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:25.810384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.804625 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx6fd\" (UniqueName: \"kubernetes.io/projected/c404f7f7-80ab-418e-acae-f76716ca7b16-kube-api-access-tx6fd\") pod \"payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k\" (UID: \"c404f7f7-80ab-418e-acae-f76716ca7b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:25.810384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.805931 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c404f7f7-80ab-418e-acae-f76716ca7b16-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k\" (UID: \"c404f7f7-80ab-418e-acae-f76716ca7b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:25.873525 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.865506 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx6fd\" (UniqueName: \"kubernetes.io/projected/c404f7f7-80ab-418e-acae-f76716ca7b16-kube-api-access-tx6fd\") pod \"payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k\" (UID: \"c404f7f7-80ab-418e-acae-f76716ca7b16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:25.909249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.909146 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7lq\" (UniqueName: \"kubernetes.io/projected/c3a0714d-13e6-4863-a6f2-79e7e883e73f-kube-api-access-zq7lq\") pod \"payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582\" (UID: \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\") " pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:25.909625 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.909597 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a0714d-13e6-4863-a6f2-79e7e883e73f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582\" (UID: \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\") " pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:25.910706 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.910681 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a0714d-13e6-4863-a6f2-79e7e883e73f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582\" (UID: \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\") " pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:25.922072 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.922041 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq7lq\" (UniqueName: \"kubernetes.io/projected/c3a0714d-13e6-4863-a6f2-79e7e883e73f-kube-api-access-zq7lq\") pod \"payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582\" (UID: \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\") " pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:25.967878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:25.967841 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:26.029834 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.029798 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:26.268510 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:48:26.268468 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc404f7f7_80ab_418e_acae_f76716ca7b16.slice/crio-4353c3a6a7f382a2cec6b1d3e9b41aecffacaa1557fc5a985fc8bd806c0f8722 WatchSource:0}: Error finding container 4353c3a6a7f382a2cec6b1d3e9b41aecffacaa1557fc5a985fc8bd806c0f8722: Status 404 returned error can't find the container with id 4353c3a6a7f382a2cec6b1d3e9b41aecffacaa1557fc5a985fc8bd806c0f8722 Nov 03 10:48:26.268510 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.268496 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k"] Nov 03 10:48:26.347373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.347344 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582"] Nov 03 10:48:26.348482 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:48:26.348451 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a0714d_13e6_4863_a6f2_79e7e883e73f.slice/crio-ce90212620e5fade3a331a6ece7ad2468142624bdd1e790d33fbce1d88386867 WatchSource:0}: Error finding container ce90212620e5fade3a331a6ece7ad2468142624bdd1e790d33fbce1d88386867: Status 404 returned error can't find the container with id ce90212620e5fade3a331a6ece7ad2468142624bdd1e790d33fbce1d88386867 Nov 03 10:48:26.641393 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.641342 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" event={"ID":"c404f7f7-80ab-418e-acae-f76716ca7b16","Type":"ContainerStarted","Data":"d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519"} Nov 03 10:48:26.641790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.641415 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" event={"ID":"c404f7f7-80ab-418e-acae-f76716ca7b16","Type":"ContainerStarted","Data":"4353c3a6a7f382a2cec6b1d3e9b41aecffacaa1557fc5a985fc8bd806c0f8722"} Nov 03 10:48:26.641790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.641448 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:26.643347 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.643286 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" event={"ID":"c3a0714d-13e6-4863-a6f2-79e7e883e73f","Type":"ContainerStarted","Data":"c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a"} Nov 03 10:48:26.643479 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.643364 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" event={"ID":"c3a0714d-13e6-4863-a6f2-79e7e883e73f","Type":"ContainerStarted","Data":"ce90212620e5fade3a331a6ece7ad2468142624bdd1e790d33fbce1d88386867"} Nov 03 10:48:26.643522 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.643490 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:26.659792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.659746 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" podStartSLOduration=1.659732709 podStartE2EDuration="1.659732709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:26.655382153 +0000 UTC m=+1184.640253436" watchObservedRunningTime="2026-11-03 10:48:26.659732709 +0000 UTC m=+1184.644603991" Nov 03 10:48:26.671562 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:26.671517 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" podStartSLOduration=1.6714949300000002 podStartE2EDuration="1.67149493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:26.669924632 +0000 UTC m=+1184.654795916" watchObservedRunningTime="2026-11-03 10:48:26.67149493 +0000 UTC m=+1184.656366271" Nov 03 10:48:31.604374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:31.604340 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:31.648824 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:31.648792 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:32.640842 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:32.640806 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:37.673630 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:37.673596 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:48:37.677794 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:37.677758 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:48:38.083381 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:38.061346 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6"] Nov 03 10:48:38.146725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:38.145857 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f"] Nov 03 10:48:39.496672 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.496640 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79"] Nov 03 10:48:39.504556 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.501883 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:39.504749 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.504588 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79"] Nov 03 10:48:39.510651 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.510534 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a51d55-dockercfg-tlwmq\"" Nov 03 10:48:39.511024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.511008 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a51d55\"" Nov 03 10:48:39.665258 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.664344 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8tps\" (UniqueName: \"kubernetes.io/projected/5877b26e-cbee-4c5f-adfc-0528c94acb92-kube-api-access-d8tps\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79\" (UID: \"5877b26e-cbee-4c5f-adfc-0528c94acb92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:39.665258 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.664702 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5877b26e-cbee-4c5f-adfc-0528c94acb92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79\" (UID: \"5877b26e-cbee-4c5f-adfc-0528c94acb92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:39.793318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.793182 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj"] Nov 03 10:48:39.856892 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.856847 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tps\" (UniqueName: \"kubernetes.io/projected/5877b26e-cbee-4c5f-adfc-0528c94acb92-kube-api-access-d8tps\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79\" (UID: \"5877b26e-cbee-4c5f-adfc-0528c94acb92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:39.857586 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.857552 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5877b26e-cbee-4c5f-adfc-0528c94acb92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79\" (UID: \"5877b26e-cbee-4c5f-adfc-0528c94acb92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:39.930569 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.930533 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5877b26e-cbee-4c5f-adfc-0528c94acb92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79\" (UID: \"5877b26e-cbee-4c5f-adfc-0528c94acb92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:39.960470 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.960007 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tps\" (UniqueName: \"kubernetes.io/projected/5877b26e-cbee-4c5f-adfc-0528c94acb92-kube-api-access-d8tps\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79\" (UID: \"5877b26e-cbee-4c5f-adfc-0528c94acb92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:39.967826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.967789 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:39.987280 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:39.984777 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj"] Nov 03 10:48:40.076352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:40.070147 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86np\" (UniqueName: \"kubernetes.io/projected/a4497595-63a7-4419-a909-569c8de96a6a-kube-api-access-t86np\") pod \"payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj\" (UID: \"a4497595-63a7-4419-a909-569c8de96a6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:40.076352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:40.070351 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4497595-63a7-4419-a909-569c8de96a6a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj\" (UID: \"a4497595-63a7-4419-a909-569c8de96a6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:40.173320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:40.172408 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4497595-63a7-4419-a909-569c8de96a6a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj\" (UID: \"a4497595-63a7-4419-a909-569c8de96a6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:40.173320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:40.172633 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86np\" (UniqueName: \"kubernetes.io/projected/a4497595-63a7-4419-a909-569c8de96a6a-kube-api-access-t86np\") pod \"payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj\" (UID: \"a4497595-63a7-4419-a909-569c8de96a6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:40.177317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:40.173840 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4497595-63a7-4419-a909-569c8de96a6a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj\" (UID: \"a4497595-63a7-4419-a909-569c8de96a6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:40.202320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:40.202053 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86np\" (UniqueName: \"kubernetes.io/projected/a4497595-63a7-4419-a909-569c8de96a6a-kube-api-access-t86np\") pod \"payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj\" (UID: \"a4497595-63a7-4419-a909-569c8de96a6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:40.231321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:40.230989 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:40.374327 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:40.373109 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:41.516360 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:41.515899 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79"] Nov 03 10:48:41.563890 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:48:41.563853 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5877b26e_cbee_4c5f_adfc_0528c94acb92.slice/crio-e803a7a80d8d6280b2eb2278ad52632e18c46200d5db77d3328cf338888069a4 WatchSource:0}: Error finding container e803a7a80d8d6280b2eb2278ad52632e18c46200d5db77d3328cf338888069a4: Status 404 returned error can't find the container with id e803a7a80d8d6280b2eb2278ad52632e18c46200d5db77d3328cf338888069a4 Nov 03 10:48:41.746675 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:41.745455 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj"] Nov 03 10:48:42.146397 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.146346 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" event={"ID":"5877b26e-cbee-4c5f-adfc-0528c94acb92","Type":"ContainerStarted","Data":"5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743"} Nov 03 10:48:42.146597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.146426 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" event={"ID":"5877b26e-cbee-4c5f-adfc-0528c94acb92","Type":"ContainerStarted","Data":"e803a7a80d8d6280b2eb2278ad52632e18c46200d5db77d3328cf338888069a4"} Nov 03 10:48:42.146597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.146461 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:42.148360 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.148333 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" event={"ID":"a4497595-63a7-4419-a909-569c8de96a6a","Type":"ContainerStarted","Data":"171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71"} Nov 03 10:48:42.148480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.148374 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" event={"ID":"a4497595-63a7-4419-a909-569c8de96a6a","Type":"ContainerStarted","Data":"146f0cb3bb4859bddd2a7ab910b8476d09902a82dc08894ce725386ddbf66447"} Nov 03 10:48:42.148480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.148394 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:42.167357 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.167312 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" podStartSLOduration=3.167284117 podStartE2EDuration="3.167284117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:42.163019002 +0000 UTC m=+1200.147890280" watchObservedRunningTime="2026-11-03 10:48:42.167284117 +0000 UTC m=+1200.152155400" Nov 03 10:48:42.189532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.189467 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" podStartSLOduration=3.189448002 podStartE2EDuration="3.189448002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:42.18536309 +0000 UTC m=+1200.170234387" watchObservedRunningTime="2026-11-03 10:48:42.189448002 +0000 UTC m=+1200.174319286" Nov 03 10:48:42.323897 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.323862 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:48:42.324731 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.324705 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:48:42.332987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.332956 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:48:42.333196 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:42.333180 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:48:43.063403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.063357 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" podUID="fde8d95b-7cdf-4686-80e9-3aeddfcbfb06" containerName="payload-processing" containerID="cri-o://0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521" gracePeriod=25 Nov 03 10:48:43.149762 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.148393 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" podUID="d26cbd8b-dd63-430d-93ce-10bb4a252a19" containerName="payload-pre-processing" containerID="cri-o://2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf" gracePeriod=25 Nov 03 10:48:43.494863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.494833 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:48:43.502852 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.502819 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-kube-api-access-qvrg8\" (UniqueName: \"kubernetes.io/projected/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-kube-api-access-qvrg8\") pod \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\" (UID: \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\") " Nov 03 10:48:43.503042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.502905 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-plugins-config-volume\") pod \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\" (UID: \"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06\") " Nov 03 10:48:43.503481 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.503440 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-plugins-config-volume" pod "fde8d95b-7cdf-4686-80e9-3aeddfcbfb06" (UID: "fde8d95b-7cdf-4686-80e9-3aeddfcbfb06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:43.505527 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.505498 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-kube-api-access-qvrg8" pod "fde8d95b-7cdf-4686-80e9-3aeddfcbfb06" (UID: "fde8d95b-7cdf-4686-80e9-3aeddfcbfb06"). InnerVolumeSpecName "kube-api-access-qvrg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:43.540209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.540180 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:48:43.604237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.604155 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26cbd8b-dd63-430d-93ce-10bb4a252a19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26cbd8b-dd63-430d-93ce-10bb4a252a19-plugins-config-volume\") pod \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\" (UID: \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\") " Nov 03 10:48:43.604537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.604326 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26cbd8b-dd63-430d-93ce-10bb4a252a19-kube-api-access-s5cfk\" (UniqueName: \"kubernetes.io/projected/d26cbd8b-dd63-430d-93ce-10bb4a252a19-kube-api-access-s5cfk\") pod \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\" (UID: \"d26cbd8b-dd63-430d-93ce-10bb4a252a19\") " Nov 03 10:48:43.604955 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.604929 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:43.605084 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.604962 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvrg8\" (UniqueName: \"kubernetes.io/projected/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06-kube-api-access-qvrg8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:43.605084 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.604924 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26cbd8b-dd63-430d-93ce-10bb4a252a19-plugins-config-volume" pod "d26cbd8b-dd63-430d-93ce-10bb4a252a19" (UID: "d26cbd8b-dd63-430d-93ce-10bb4a252a19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:43.608339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.607897 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26cbd8b-dd63-430d-93ce-10bb4a252a19-kube-api-access-s5cfk" pod "d26cbd8b-dd63-430d-93ce-10bb4a252a19" (UID: "d26cbd8b-dd63-430d-93ce-10bb4a252a19"). InnerVolumeSpecName "kube-api-access-s5cfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:43.706553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.706517 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26cbd8b-dd63-430d-93ce-10bb4a252a19-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:43.706553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:43.706547 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5cfk\" (UniqueName: \"kubernetes.io/projected/d26cbd8b-dd63-430d-93ce-10bb4a252a19-kube-api-access-s5cfk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:44.205823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.205778 5086 generic.go:356] "Generic (PLEG): container finished" podID="d26cbd8b-dd63-430d-93ce-10bb4a252a19" containerID="2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf" exitCode=0 Nov 03 10:48:44.206379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.206025 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" Nov 03 10:48:44.206379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.206181 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" event={"ID":"d26cbd8b-dd63-430d-93ce-10bb4a252a19","Type":"ContainerDied","Data":"2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf"} Nov 03 10:48:44.206379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.206259 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f" event={"ID":"d26cbd8b-dd63-430d-93ce-10bb4a252a19","Type":"ContainerDied","Data":"2495ca242b4f98e394ac8b4e405727ab8ddedbc280b0231c465d8f59e158e5da"} Nov 03 10:48:44.206379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.206289 5086 scope.go:122] "RemoveContainer" containerID="2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf" Nov 03 10:48:44.209840 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.209814 5086 generic.go:356] "Generic (PLEG): container finished" podID="fde8d95b-7cdf-4686-80e9-3aeddfcbfb06" containerID="0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521" exitCode=0 Nov 03 10:48:44.210130 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.210103 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" event={"ID":"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06","Type":"ContainerDied","Data":"0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521"} Nov 03 10:48:44.210314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.210281 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" event={"ID":"fde8d95b-7cdf-4686-80e9-3aeddfcbfb06","Type":"ContainerDied","Data":"5c376749bd84895a3ca5782286004576b760b99877e2fea8b7d3ba1393a2a9b4"} Nov 03 10:48:44.210923 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.210442 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6" Nov 03 10:48:44.227911 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.227882 5086 scope.go:122] "RemoveContainer" containerID="2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf" Nov 03 10:48:44.233485 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:44.233450 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf\": container with ID starting with 2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf not found: ID does not exist" containerID="2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf" Nov 03 10:48:44.233630 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.233497 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf"} err="failed to get container status \"2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf\": rpc error: code = NotFound desc = could not find container \"2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf\": container with ID starting with 2086cc4c0ac2d45957cf35eaaebefbd707897f9573cfbfedced27b4f8b9690bf not found: ID does not exist" Nov 03 10:48:44.233630 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.233523 5086 scope.go:122] "RemoveContainer" containerID="0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521" Nov 03 10:48:44.249620 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.249439 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6"] Nov 03 10:48:44.249719 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.249651 5086 scope.go:122] "RemoveContainer" containerID="0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521" Nov 03 10:48:44.250020 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:44.249996 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521\": container with ID starting with 0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521 not found: ID does not exist" containerID="0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521" Nov 03 10:48:44.250136 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.250023 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521"} err="failed to get container status \"0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521\": rpc error: code = NotFound desc = could not find container \"0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521\": container with ID starting with 0584860b6009f7b06ab1cb6214b19a004ad5e39cad1c0980c5cb620ea33c4521 not found: ID does not exist" Nov 03 10:48:44.273828 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.273784 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-642fe5-dd986566f-m6bd6"] Nov 03 10:48:44.307510 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.307470 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f"] Nov 03 10:48:44.317957 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:44.317921 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-642fe5-6bb9668fc9-4dc6f"] Nov 03 10:48:46.174559 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.174526 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26cbd8b-dd63-430d-93ce-10bb4a252a19" path="/var/lib/kubelet/pods/d26cbd8b-dd63-430d-93ce-10bb4a252a19/volumes" Nov 03 10:48:46.175084 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.175069 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde8d95b-7cdf-4686-80e9-3aeddfcbfb06" path="/var/lib/kubelet/pods/fde8d95b-7cdf-4686-80e9-3aeddfcbfb06/volumes" Nov 03 10:48:46.725545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.725508 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght"] Nov 03 10:48:46.726666 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.726642 5086 cpu_manager.go:414] "containerMap: removing container" podUID="fde8d95b-7cdf-4686-80e9-3aeddfcbfb06" containerName="payload-processing" Nov 03 10:48:46.726666 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.726664 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde8d95b-7cdf-4686-80e9-3aeddfcbfb06" containerName="payload-processing" Nov 03 10:48:46.726890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.726696 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d26cbd8b-dd63-430d-93ce-10bb4a252a19" containerName="payload-pre-processing" Nov 03 10:48:46.726890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.726705 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26cbd8b-dd63-430d-93ce-10bb4a252a19" containerName="payload-pre-processing" Nov 03 10:48:46.726890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.726848 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde8d95b-7cdf-4686-80e9-3aeddfcbfb06" containerName="payload-processing" Nov 03 10:48:46.726890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.726865 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26cbd8b-dd63-430d-93ce-10bb4a252a19" containerName="payload-pre-processing" Nov 03 10:48:46.765811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.765496 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.772478 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.772384 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght"] Nov 03 10:48:46.776327 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.775661 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a51d55-serving-cert\"" Nov 03 10:48:46.873378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.873334 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-metrics-certs\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.873549 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.873413 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdpgr\" (UniqueName: \"kubernetes.io/projected/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-kube-api-access-qdpgr\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.873654 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.873628 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.873937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.873915 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.975066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.975031 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-metrics-certs\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.975241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.975094 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdpgr\" (UniqueName: \"kubernetes.io/projected/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-kube-api-access-qdpgr\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.975241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.975156 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.975388 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.975263 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.976088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.976015 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.978034 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.978002 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.978144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.978108 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-metrics-certs\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:46.982883 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:46.982858 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdpgr\" (UniqueName: \"kubernetes.io/projected/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-kube-api-access-qdpgr\") pod \"maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:47.103267 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:47.103232 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:47.399981 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:47.399944 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght"] Nov 03 10:48:47.406344 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:48:47.405819 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6e6ee0_b1f0_44a8_aaaa_7e08e90c08b5.slice/crio-c014d78eac9f4f414adc8a931553e741b09d6afedc3b998b8a6eceb0ce8748fd WatchSource:0}: Error finding container c014d78eac9f4f414adc8a931553e741b09d6afedc3b998b8a6eceb0ce8748fd: Status 404 returned error can't find the container with id c014d78eac9f4f414adc8a931553e741b09d6afedc3b998b8a6eceb0ce8748fd Nov 03 10:48:48.251739 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:48.251653 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" event={"ID":"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5","Type":"ContainerStarted","Data":"836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f"} Nov 03 10:48:48.251739 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:48.251709 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" event={"ID":"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5","Type":"ContainerStarted","Data":"c014d78eac9f4f414adc8a931553e741b09d6afedc3b998b8a6eceb0ce8748fd"} Nov 03 10:48:48.251941 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:48.251873 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:48.273339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:48.273268 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" podStartSLOduration=1.816540726 podStartE2EDuration="2.27325334s" totalImagesPullingTime="456.712614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:48.27018958 +0000 UTC m=+1206.255060863" watchObservedRunningTime="2026-11-03 10:48:48.27325334 +0000 UTC m=+1206.258124623" Nov 03 10:48:50.469992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.469961 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh"] Nov 03 10:48:50.474553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.474534 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:48:50.503371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.503338 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-38c9b0\"" Nov 03 10:48:50.504047 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.503892 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-38c9b0-dockercfg-pxv8p\"" Nov 03 10:48:50.507006 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.506981 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh"] Nov 03 10:48:50.527905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.527866 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nxq\" (UniqueName: \"kubernetes.io/projected/9feef9dd-d6c6-4cd0-8928-259d28809ce1-kube-api-access-49nxq\") pod \"payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh\" (UID: \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:48:50.530275 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.527976 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feef9dd-d6c6-4cd0-8928-259d28809ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh\" (UID: \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:48:50.649317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.637261 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nxq\" (UniqueName: \"kubernetes.io/projected/9feef9dd-d6c6-4cd0-8928-259d28809ce1-kube-api-access-49nxq\") pod \"payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh\" (UID: \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:48:50.649317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.637383 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feef9dd-d6c6-4cd0-8928-259d28809ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh\" (UID: \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:48:50.649317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.638186 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feef9dd-d6c6-4cd0-8928-259d28809ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh\" (UID: \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:48:50.675333 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.665673 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj"] Nov 03 10:48:50.679676 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.676565 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj"] Nov 03 10:48:50.679676 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.676687 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:48:50.702316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.698956 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nxq\" (UniqueName: \"kubernetes.io/projected/9feef9dd-d6c6-4cd0-8928-259d28809ce1-kube-api-access-49nxq\") pod \"payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh\" (UID: \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:48:50.790650 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.790558 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:48:50.841827 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.841397 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3692a26a-a911-44ea-b9fa-3c91af5d7748-plugins-config-volume\") pod \"payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj\" (UID: \"3692a26a-a911-44ea-b9fa-3c91af5d7748\") " pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:48:50.841827 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.841549 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98t9\" (UniqueName: \"kubernetes.io/projected/3692a26a-a911-44ea-b9fa-3c91af5d7748-kube-api-access-c98t9\") pod \"payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj\" (UID: \"3692a26a-a911-44ea-b9fa-3c91af5d7748\") " pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:48:50.942689 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.942646 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3692a26a-a911-44ea-b9fa-3c91af5d7748-plugins-config-volume\") pod \"payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj\" (UID: \"3692a26a-a911-44ea-b9fa-3c91af5d7748\") " pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:48:50.942885 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.942781 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98t9\" (UniqueName: \"kubernetes.io/projected/3692a26a-a911-44ea-b9fa-3c91af5d7748-kube-api-access-c98t9\") pod \"payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj\" (UID: \"3692a26a-a911-44ea-b9fa-3c91af5d7748\") " pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:48:50.943448 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.943424 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3692a26a-a911-44ea-b9fa-3c91af5d7748-plugins-config-volume\") pod \"payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj\" (UID: \"3692a26a-a911-44ea-b9fa-3c91af5d7748\") " pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:48:50.950488 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:50.950462 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98t9\" (UniqueName: \"kubernetes.io/projected/3692a26a-a911-44ea-b9fa-3c91af5d7748-kube-api-access-c98t9\") pod \"payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj\" (UID: \"3692a26a-a911-44ea-b9fa-3c91af5d7748\") " pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:48:51.022119 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:51.022084 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:48:51.112036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:51.103909 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh"] Nov 03 10:48:51.115702 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:48:51.115668 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9feef9dd_d6c6_4cd0_8928_259d28809ce1.slice/crio-192e2ac96d33c92007c041d066b5793b79d75252cac6d0d81ca0c6d2ff6b843a WatchSource:0}: Error finding container 192e2ac96d33c92007c041d066b5793b79d75252cac6d0d81ca0c6d2ff6b843a: Status 404 returned error can't find the container with id 192e2ac96d33c92007c041d066b5793b79d75252cac6d0d81ca0c6d2ff6b843a Nov 03 10:48:51.292045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:51.292004 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" event={"ID":"9feef9dd-d6c6-4cd0-8928-259d28809ce1","Type":"ContainerStarted","Data":"57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66"} Nov 03 10:48:51.292248 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:51.292083 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" event={"ID":"9feef9dd-d6c6-4cd0-8928-259d28809ce1","Type":"ContainerStarted","Data":"192e2ac96d33c92007c041d066b5793b79d75252cac6d0d81ca0c6d2ff6b843a"} Nov 03 10:48:51.293258 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:51.293236 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:48:51.319333 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:48:51.319264 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3692a26a_a911_44ea_b9fa_3c91af5d7748.slice/crio-c364f605eecc25f7670338efdd9ff34a60920efa51ef3a5855f534a789c18b66 WatchSource:0}: Error finding container c364f605eecc25f7670338efdd9ff34a60920efa51ef3a5855f534a789c18b66: Status 404 returned error can't find the container with id c364f605eecc25f7670338efdd9ff34a60920efa51ef3a5855f534a789c18b66 Nov 03 10:48:51.321347 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:51.321271 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" podStartSLOduration=1.321251616 podStartE2EDuration="1.321251616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:51.313575979 +0000 UTC m=+1209.298447259" watchObservedRunningTime="2026-11-03 10:48:51.321251616 +0000 UTC m=+1209.306122900" Nov 03 10:48:51.328358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:51.328315 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj"] Nov 03 10:48:52.300917 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:52.300868 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" event={"ID":"3692a26a-a911-44ea-b9fa-3c91af5d7748","Type":"ContainerStarted","Data":"21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd"} Nov 03 10:48:52.301405 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:52.300944 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" event={"ID":"3692a26a-a911-44ea-b9fa-3c91af5d7748","Type":"ContainerStarted","Data":"c364f605eecc25f7670338efdd9ff34a60920efa51ef3a5855f534a789c18b66"} Nov 03 10:48:52.319377 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:52.319326 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" podStartSLOduration=2.319311327 podStartE2EDuration="2.319311327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:52.315978025 +0000 UTC m=+1210.300849310" watchObservedRunningTime="2026-11-03 10:48:52.319311327 +0000 UTC m=+1210.304182604" Nov 03 10:48:53.193852 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:53.193631 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:48:53.193852 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:53.193704 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:48:53.309596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:53.309562 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:48:53.549984 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:53.549900 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz"] Nov 03 10:48:53.550169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:53.550142 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" podUID="1db645df-be5b-4543-99db-ddb4f498b141" containerName="maas-api" containerID="cri-o://e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3" gracePeriod=30 Nov 03 10:48:53.737869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:53.737816 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" podUID="1db645df-be5b-4543-99db-ddb4f498b141" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.110:8443/health\": dial tcp 10.132.0.110:8443: connect: connection refused" Nov 03 10:48:54.017721 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.017690 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:48:54.058830 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.058789 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-metrics-certs\") pod \"1db645df-be5b-4543-99db-ddb4f498b141\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " Nov 03 10:48:54.064817 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.064784 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-metrics-certs" pod "1db645df-be5b-4543-99db-ddb4f498b141" (UID: "1db645df-be5b-4543-99db-ddb4f498b141"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:48:54.108785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.108744 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg"] Nov 03 10:48:54.121458 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.121284 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn"] Nov 03 10:48:54.161346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.160860 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-maas-api-tls\") pod \"1db645df-be5b-4543-99db-ddb4f498b141\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " Nov 03 10:48:54.162050 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.162021 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db645df-be5b-4543-99db-ddb4f498b141-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db645df-be5b-4543-99db-ddb4f498b141-openshift-service-ca\") pod \"1db645df-be5b-4543-99db-ddb4f498b141\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " Nov 03 10:48:54.165225 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.162811 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db645df-be5b-4543-99db-ddb4f498b141-openshift-service-ca" pod "1db645df-be5b-4543-99db-ddb4f498b141" (UID: "1db645df-be5b-4543-99db-ddb4f498b141"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:54.189407 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.189373 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-maas-api-tls" pod "1db645df-be5b-4543-99db-ddb4f498b141" (UID: "1db645df-be5b-4543-99db-ddb4f498b141"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:48:54.189671 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.189651 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db645df-be5b-4543-99db-ddb4f498b141-kube-api-access-78n4f\" (UniqueName: \"kubernetes.io/projected/1db645df-be5b-4543-99db-ddb4f498b141-kube-api-access-78n4f\") pod \"1db645df-be5b-4543-99db-ddb4f498b141\" (UID: \"1db645df-be5b-4543-99db-ddb4f498b141\") " Nov 03 10:48:54.190913 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.190889 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:54.191049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.191034 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db645df-be5b-4543-99db-ddb4f498b141-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:54.191359 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.191341 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db645df-be5b-4543-99db-ddb4f498b141-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:54.203012 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.195560 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db645df-be5b-4543-99db-ddb4f498b141-kube-api-access-78n4f" pod "1db645df-be5b-4543-99db-ddb4f498b141" (UID: "1db645df-be5b-4543-99db-ddb4f498b141"). InnerVolumeSpecName "kube-api-access-78n4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:54.266784 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.266704 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:48:54.287332 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:54.287257 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:48:54.294117 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.294084 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78n4f\" (UniqueName: \"kubernetes.io/projected/1db645df-be5b-4543-99db-ddb4f498b141-kube-api-access-78n4f\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:54.319499 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.319457 5086 generic.go:356] "Generic (PLEG): container finished" podID="1db645df-be5b-4543-99db-ddb4f498b141" containerID="e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3" exitCode=0 Nov 03 10:48:54.320022 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.319507 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" Nov 03 10:48:54.320022 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.319899 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" event={"ID":"1db645df-be5b-4543-99db-ddb4f498b141","Type":"ContainerDied","Data":"e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3"} Nov 03 10:48:54.320022 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.319980 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz" event={"ID":"1db645df-be5b-4543-99db-ddb4f498b141","Type":"ContainerDied","Data":"b107ffa3ce4531b940ab9ffdfa1a0da73275f5a8e510d230f959d678ae3399e6"} Nov 03 10:48:54.320022 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.320016 5086 scope.go:122] "RemoveContainer" containerID="e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3" Nov 03 10:48:54.332560 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.332537 5086 scope.go:122] "RemoveContainer" containerID="e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3" Nov 03 10:48:54.332879 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:54.332857 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3\": container with ID starting with e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3 not found: ID does not exist" containerID="e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3" Nov 03 10:48:54.332974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.332885 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3"} err="failed to get container status \"e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3\": rpc error: code = NotFound desc = could not find container \"e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3\": container with ID starting with e04def52cb44516b22b1c0f7af472d13ba8565b6cf969c7de2a84f855ed000b3 not found: ID does not exist" Nov 03 10:48:54.344480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.344444 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz"] Nov 03 10:48:54.347822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:54.347793 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-642fe5-7f449c85fd-cpjgz"] Nov 03 10:48:56.172631 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.172595 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db645df-be5b-4543-99db-ddb4f498b141" path="/var/lib/kubelet/pods/1db645df-be5b-4543-99db-ddb4f498b141/volumes" Nov 03 10:48:56.659746 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.659713 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh"] Nov 03 10:48:56.660520 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.660504 5086 cpu_manager.go:414] "containerMap: removing container" podUID="1db645df-be5b-4543-99db-ddb4f498b141" containerName="maas-api" Nov 03 10:48:56.660596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.660522 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db645df-be5b-4543-99db-ddb4f498b141" containerName="maas-api" Nov 03 10:48:56.660638 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.660629 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db645df-be5b-4543-99db-ddb4f498b141" containerName="maas-api" Nov 03 10:48:56.677831 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.677803 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh"] Nov 03 10:48:56.678261 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.678190 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.686796 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.686764 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-38c9b0-serving-cert\"" Nov 03 10:48:56.757343 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.757156 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xfv\" (UniqueName: \"kubernetes.io/projected/ef6d124d-4f64-41fc-ad08-a3878a62da29-kube-api-access-h6xfv\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.757753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.757725 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-maas-api-tls\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.757882 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.757865 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6d124d-4f64-41fc-ad08-a3878a62da29-openshift-service-ca\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.757945 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.757929 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-metrics-certs\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.860569 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.860530 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-maas-api-tls\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.861854 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.861825 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6d124d-4f64-41fc-ad08-a3878a62da29-openshift-service-ca\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.862891 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.862872 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-metrics-certs\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.863960 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.863920 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xfv\" (UniqueName: \"kubernetes.io/projected/ef6d124d-4f64-41fc-ad08-a3878a62da29-kube-api-access-h6xfv\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.878512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.862803 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6d124d-4f64-41fc-ad08-a3878a62da29-openshift-service-ca\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.878512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.870469 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-maas-api-tls\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.878512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.875829 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xfv\" (UniqueName: \"kubernetes.io/projected/ef6d124d-4f64-41fc-ad08-a3878a62da29-kube-api-access-h6xfv\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.885876 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.882253 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-metrics-certs\") pod \"maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:56.999846 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:56.999747 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:57.303961 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:57.303933 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh"] Nov 03 10:48:57.305085 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:48:57.305057 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6d124d_4f64_41fc_ad08_a3878a62da29.slice/crio-5e8f3c9e9e25d7687e8cb847a7f4c9ef10370a92ed18456bc0e4f32fcf1c73e4 WatchSource:0}: Error finding container 5e8f3c9e9e25d7687e8cb847a7f4c9ef10370a92ed18456bc0e4f32fcf1c73e4: Status 404 returned error can't find the container with id 5e8f3c9e9e25d7687e8cb847a7f4c9ef10370a92ed18456bc0e4f32fcf1c73e4 Nov 03 10:48:57.433488 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:57.433452 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" event={"ID":"ef6d124d-4f64-41fc-ad08-a3878a62da29","Type":"ContainerStarted","Data":"5e8f3c9e9e25d7687e8cb847a7f4c9ef10370a92ed18456bc0e4f32fcf1c73e4"} Nov 03 10:48:57.854069 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:57.854023 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj"] Nov 03 10:48:57.874411 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:57.874358 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79"] Nov 03 10:48:58.444257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:58.444220 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" event={"ID":"ef6d124d-4f64-41fc-ad08-a3878a62da29","Type":"ContainerStarted","Data":"9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756"} Nov 03 10:48:58.444724 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:58.444343 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:48:58.462781 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:58.462731 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" podStartSLOduration=2.051750849 podStartE2EDuration="2.462714561s" totalImagesPullingTime="410.963712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:58.459390322 +0000 UTC m=+1216.444261606" watchObservedRunningTime="2026-11-03 10:48:58.462714561 +0000 UTC m=+1216.447585844" Nov 03 10:48:58.692050 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:58.692006 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml"] Nov 03 10:48:58.693191 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:58.693151 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" podUID="be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" containerName="maas-api" containerID="cri-o://92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f" gracePeriod=30 Nov 03 10:48:59.035447 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.035392 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:59.038675 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.038651 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-maas-api-tls\") pod \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " Nov 03 10:48:59.038853 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.038696 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-openshift-service-ca\") pod \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " Nov 03 10:48:59.038853 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.038776 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-kube-api-access-zxmfj\" (UniqueName: \"kubernetes.io/projected/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-kube-api-access-zxmfj\") pod \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " Nov 03 10:48:59.038957 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.038861 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-metrics-certs\") pod \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\" (UID: \"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d\") " Nov 03 10:48:59.039887 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.039837 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-openshift-service-ca" pod "be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" (UID: "be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:59.041494 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.041465 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-maas-api-tls" pod "be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" (UID: "be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:48:59.042624 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.042594 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-kube-api-access-zxmfj" pod "be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" (UID: "be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d"). InnerVolumeSpecName "kube-api-access-zxmfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:59.044085 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.044064 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-metrics-certs" pod "be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" (UID: "be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:48:59.109955 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.109771 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" podUID="3fe0f981-a314-4e19-9d4b-cce9b3c08e6e" containerName="payload-processing" containerID="cri-o://2a575e6ad82300753acce0554eab809d6a0cf9dd2532056a6ad4f7e48383d545" gracePeriod=25 Nov 03 10:48:59.122118 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.122056 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" podUID="7e5648eb-b433-40b1-bc9b-b1d1f7106d81" containerName="payload-pre-processing" containerID="cri-o://a84e1d47c3be80eaccd57b3571a7d6bf80a7945ef4aa9443ada5df45a60bc0bd" gracePeriod=25 Nov 03 10:48:59.140467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.140439 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:59.140467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.140467 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:59.140626 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.140477 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxmfj\" (UniqueName: \"kubernetes.io/projected/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-kube-api-access-zxmfj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:59.140626 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.140495 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:59.457604 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.457575 5086 generic.go:356] "Generic (PLEG): container finished" podID="7e5648eb-b433-40b1-bc9b-b1d1f7106d81" containerID="a84e1d47c3be80eaccd57b3571a7d6bf80a7945ef4aa9443ada5df45a60bc0bd" exitCode=0 Nov 03 10:48:59.458018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.457633 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" event={"ID":"7e5648eb-b433-40b1-bc9b-b1d1f7106d81","Type":"ContainerDied","Data":"a84e1d47c3be80eaccd57b3571a7d6bf80a7945ef4aa9443ada5df45a60bc0bd"} Nov 03 10:48:59.458708 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.458689 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:59.459207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.459185 5086 generic.go:356] "Generic (PLEG): container finished" podID="be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" containerID="92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f" exitCode=0 Nov 03 10:48:59.459330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.459235 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" Nov 03 10:48:59.459330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.459267 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" event={"ID":"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d","Type":"ContainerDied","Data":"92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f"} Nov 03 10:48:59.459462 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.459361 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml" event={"ID":"be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d","Type":"ContainerDied","Data":"06212282dceab04266f047c9b1d41fbf7c6622e58b6c549a941c2485bc5a57e2"} Nov 03 10:48:59.459462 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.459398 5086 scope.go:122] "RemoveContainer" containerID="92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f" Nov 03 10:48:59.461049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.460916 5086 generic.go:356] "Generic (PLEG): container finished" podID="3fe0f981-a314-4e19-9d4b-cce9b3c08e6e" containerID="2a575e6ad82300753acce0554eab809d6a0cf9dd2532056a6ad4f7e48383d545" exitCode=0 Nov 03 10:48:59.461049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.460968 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" Nov 03 10:48:59.461049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.460993 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg" event={"ID":"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e","Type":"ContainerDied","Data":"2a575e6ad82300753acce0554eab809d6a0cf9dd2532056a6ad4f7e48383d545"} Nov 03 10:48:59.497539 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.497507 5086 scope.go:122] "RemoveContainer" containerID="92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f" Nov 03 10:48:59.498022 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:48:59.497992 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f\": container with ID starting with 92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f not found: ID does not exist" containerID="92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f" Nov 03 10:48:59.498129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.498031 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f"} err="failed to get container status \"92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f\": rpc error: code = NotFound desc = could not find container \"92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f\": container with ID starting with 92c796402dea65e4ac3a6cfed73cb944f17903bd781e755f55153b8557d6768f not found: ID does not exist" Nov 03 10:48:59.498129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.498057 5086 scope.go:122] "RemoveContainer" containerID="2a575e6ad82300753acce0554eab809d6a0cf9dd2532056a6ad4f7e48383d545" Nov 03 10:48:59.502596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.502567 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml"] Nov 03 10:48:59.507086 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.507055 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab592834-8448b97db-h2fml"] Nov 03 10:48:59.547436 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.547411 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:48:59.548785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.548762 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-plugins-config-volume\") pod \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\" (UID: \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\") " Nov 03 10:48:59.548966 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.548951 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-kube-api-access-ccff6\") pod \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\" (UID: \"3fe0f981-a314-4e19-9d4b-cce9b3c08e6e\") " Nov 03 10:48:59.549092 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.549072 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-plugins-config-volume" pod "3fe0f981-a314-4e19-9d4b-cce9b3c08e6e" (UID: "3fe0f981-a314-4e19-9d4b-cce9b3c08e6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:59.549662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.549642 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:59.550943 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.550923 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-kube-api-access-ccff6" pod "3fe0f981-a314-4e19-9d4b-cce9b3c08e6e" (UID: "3fe0f981-a314-4e19-9d4b-cce9b3c08e6e"). InnerVolumeSpecName "kube-api-access-ccff6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:59.650235 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.650203 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-kube-api-access-7g6hf\" (UniqueName: \"kubernetes.io/projected/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-kube-api-access-7g6hf\") pod \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\" (UID: \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\") " Nov 03 10:48:59.650463 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.650308 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-plugins-config-volume\") pod \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\" (UID: \"7e5648eb-b433-40b1-bc9b-b1d1f7106d81\") " Nov 03 10:48:59.650735 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.650711 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-plugins-config-volume" pod "7e5648eb-b433-40b1-bc9b-b1d1f7106d81" (UID: "7e5648eb-b433-40b1-bc9b-b1d1f7106d81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:48:59.650893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.650878 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e-kube-api-access-ccff6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:59.650957 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.650899 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:59.652403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.652381 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-kube-api-access-7g6hf" pod "7e5648eb-b433-40b1-bc9b-b1d1f7106d81" (UID: "7e5648eb-b433-40b1-bc9b-b1d1f7106d81"). InnerVolumeSpecName "kube-api-access-7g6hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:48:59.751984 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.751887 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g6hf\" (UniqueName: \"kubernetes.io/projected/7e5648eb-b433-40b1-bc9b-b1d1f7106d81-kube-api-access-7g6hf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:48:59.786562 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.786519 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg"] Nov 03 10:48:59.790155 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:48:59.790130 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ab592834-6f7b8f6bc5-qswwg"] Nov 03 10:49:00.172802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:00.172774 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe0f981-a314-4e19-9d4b-cce9b3c08e6e" path="/var/lib/kubelet/pods/3fe0f981-a314-4e19-9d4b-cce9b3c08e6e/volumes" Nov 03 10:49:00.173252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:00.173238 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" path="/var/lib/kubelet/pods/be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d/volumes" Nov 03 10:49:00.469661 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:00.469577 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" Nov 03 10:49:00.469661 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:00.469620 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn" event={"ID":"7e5648eb-b433-40b1-bc9b-b1d1f7106d81","Type":"ContainerDied","Data":"3417c7b4c3bd3b538c4cfd649c4c5695530eaa133835b289f732613e7923f226"} Nov 03 10:49:00.470195 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:00.469678 5086 scope.go:122] "RemoveContainer" containerID="a84e1d47c3be80eaccd57b3571a7d6bf80a7945ef4aa9443ada5df45a60bc0bd" Nov 03 10:49:00.487796 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:00.487768 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn"] Nov 03 10:49:00.491372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:00.491351 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab592834-7b89945cbc-fwktn"] Nov 03 10:49:02.172232 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:02.172202 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5648eb-b433-40b1-bc9b-b1d1f7106d81" path="/var/lib/kubelet/pods/7e5648eb-b433-40b1-bc9b-b1d1f7106d81/volumes" Nov 03 10:49:02.855152 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:02.855085 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" podUID="a4497595-63a7-4419-a909-569c8de96a6a" containerName="payload-processing" containerID="cri-o://171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71" gracePeriod=25 Nov 03 10:49:02.875787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:02.875752 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" podUID="5877b26e-cbee-4c5f-adfc-0528c94acb92" containerName="payload-pre-processing" containerID="cri-o://5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743" gracePeriod=25 Nov 03 10:49:03.188382 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.188354 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:49:03.242859 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.242826 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:49:03.313025 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.312994 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:49:03.349203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.349165 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5877b26e-cbee-4c5f-adfc-0528c94acb92-kube-api-access-d8tps\" (UniqueName: \"kubernetes.io/projected/5877b26e-cbee-4c5f-adfc-0528c94acb92-kube-api-access-d8tps\") pod \"5877b26e-cbee-4c5f-adfc-0528c94acb92\" (UID: \"5877b26e-cbee-4c5f-adfc-0528c94acb92\") " Nov 03 10:49:03.349397 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.349264 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5877b26e-cbee-4c5f-adfc-0528c94acb92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5877b26e-cbee-4c5f-adfc-0528c94acb92-plugins-config-volume\") pod \"5877b26e-cbee-4c5f-adfc-0528c94acb92\" (UID: \"5877b26e-cbee-4c5f-adfc-0528c94acb92\") " Nov 03 10:49:03.349693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.349661 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5877b26e-cbee-4c5f-adfc-0528c94acb92-plugins-config-volume" pod "5877b26e-cbee-4c5f-adfc-0528c94acb92" (UID: "5877b26e-cbee-4c5f-adfc-0528c94acb92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:03.349976 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.349952 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4497595-63a7-4419-a909-569c8de96a6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4497595-63a7-4419-a909-569c8de96a6a-plugins-config-volume\") pod \"a4497595-63a7-4419-a909-569c8de96a6a\" (UID: \"a4497595-63a7-4419-a909-569c8de96a6a\") " Nov 03 10:49:03.371871 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.370483 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4497595-63a7-4419-a909-569c8de96a6a-kube-api-access-t86np\" (UniqueName: \"kubernetes.io/projected/a4497595-63a7-4419-a909-569c8de96a6a-kube-api-access-t86np\") pod \"a4497595-63a7-4419-a909-569c8de96a6a\" (UID: \"a4497595-63a7-4419-a909-569c8de96a6a\") " Nov 03 10:49:03.371871 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.371285 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5877b26e-cbee-4c5f-adfc-0528c94acb92-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:03.373943 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.373908 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5877b26e-cbee-4c5f-adfc-0528c94acb92-kube-api-access-d8tps" pod "5877b26e-cbee-4c5f-adfc-0528c94acb92" (UID: "5877b26e-cbee-4c5f-adfc-0528c94acb92"). InnerVolumeSpecName "kube-api-access-d8tps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:03.375979 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.375928 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4497595-63a7-4419-a909-569c8de96a6a-plugins-config-volume" pod "a4497595-63a7-4419-a909-569c8de96a6a" (UID: "a4497595-63a7-4419-a909-569c8de96a6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:03.458804 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.458720 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4497595-63a7-4419-a909-569c8de96a6a-kube-api-access-t86np" pod "a4497595-63a7-4419-a909-569c8de96a6a" (UID: "a4497595-63a7-4419-a909-569c8de96a6a"). InnerVolumeSpecName "kube-api-access-t86np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:03.552041 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.551994 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4497595-63a7-4419-a909-569c8de96a6a-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:03.552041 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.552041 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86np\" (UniqueName: \"kubernetes.io/projected/a4497595-63a7-4419-a909-569c8de96a6a-kube-api-access-t86np\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:03.552275 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.552066 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8tps\" (UniqueName: \"kubernetes.io/projected/5877b26e-cbee-4c5f-adfc-0528c94acb92-kube-api-access-d8tps\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:03.586323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.572590 5086 generic.go:356] "Generic (PLEG): container finished" podID="5877b26e-cbee-4c5f-adfc-0528c94acb92" containerID="5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743" exitCode=0 Nov 03 10:49:03.586323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.572644 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" event={"ID":"5877b26e-cbee-4c5f-adfc-0528c94acb92","Type":"ContainerDied","Data":"5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743"} Nov 03 10:49:03.586323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.572673 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" Nov 03 10:49:03.586323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.572720 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" event={"ID":"5877b26e-cbee-4c5f-adfc-0528c94acb92","Type":"ContainerDied","Data":"e803a7a80d8d6280b2eb2278ad52632e18c46200d5db77d3328cf338888069a4"} Nov 03 10:49:03.586323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.572752 5086 scope.go:122] "RemoveContainer" containerID="5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743" Nov 03 10:49:03.586323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.575696 5086 generic.go:356] "Generic (PLEG): container finished" podID="a4497595-63a7-4419-a909-569c8de96a6a" containerID="171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71" exitCode=0 Nov 03 10:49:03.586323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.575894 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" event={"ID":"a4497595-63a7-4419-a909-569c8de96a6a","Type":"ContainerDied","Data":"171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71"} Nov 03 10:49:03.586323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.575940 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" event={"ID":"a4497595-63a7-4419-a909-569c8de96a6a","Type":"ContainerDied","Data":"146f0cb3bb4859bddd2a7ab910b8476d09902a82dc08894ce725386ddbf66447"} Nov 03 10:49:03.586323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.576010 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj" Nov 03 10:49:03.590935 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.589802 5086 scope.go:122] "RemoveContainer" containerID="5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743" Nov 03 10:49:03.591884 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:03.591854 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743\": container with ID starting with 5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743 not found: ID does not exist" containerID="5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743" Nov 03 10:49:03.592024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.591894 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743"} err="failed to get container status \"5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743\": rpc error: code = NotFound desc = could not find container \"5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743\": container with ID starting with 5e5905960dc1613ccbd61bd567610ffb0d7f0f1be0c3b75eb17c0225d0c84743 not found: ID does not exist" Nov 03 10:49:03.592024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.591918 5086 scope.go:122] "RemoveContainer" containerID="171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71" Nov 03 10:49:03.611373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.609947 5086 scope.go:122] "RemoveContainer" containerID="171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71" Nov 03 10:49:03.611373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.610083 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79"] Nov 03 10:49:03.611373 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:03.610661 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71\": container with ID starting with 171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71 not found: ID does not exist" containerID="171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71" Nov 03 10:49:03.611373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.610718 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71"} err="failed to get container status \"171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71\": rpc error: code = NotFound desc = could not find container \"171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71\": container with ID starting with 171786cea3468e6092b80aeb3fb8ee7dd28d2ff67ffc2f93f5560debd4274c71 not found: ID does not exist" Nov 03 10:49:03.617389 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.617358 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79"] Nov 03 10:49:03.626438 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.626410 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj"] Nov 03 10:49:03.632134 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:03.632108 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-cb498f7d7-ppmlj"] Nov 03 10:49:04.177037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:04.176955 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5877b26e-cbee-4c5f-adfc-0528c94acb92" path="/var/lib/kubelet/pods/5877b26e-cbee-4c5f-adfc-0528c94acb92/volumes" Nov 03 10:49:04.177544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:04.177523 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4497595-63a7-4419-a909-569c8de96a6a" path="/var/lib/kubelet/pods/a4497595-63a7-4419-a909-569c8de96a6a/volumes" Nov 03 10:49:04.189568 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:04.189514 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-d9996b99-vct79" podUID="5877b26e-cbee-4c5f-adfc-0528c94acb92" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.116:50052\" within 1s: context deadline exceeded" Nov 03 10:49:04.190496 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:04.189582 5086 logging.go:55] [core] [Channel #1076 SubChannel #1077]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.116:50052", ServerName: "10.132.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.116:50052: operation was canceled" Nov 03 10:49:04.324373 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:04.324339 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:49:04.469493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:04.468585 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" podUID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.121:8443/health\": dial tcp 10.132.0.121:8443: connect: connection refused" Nov 03 10:49:05.150335 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.150284 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27"] Nov 03 10:49:05.150987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.150973 5086 cpu_manager.go:414] "containerMap: removing container" podUID="7e5648eb-b433-40b1-bc9b-b1d1f7106d81" containerName="payload-pre-processing" Nov 03 10:49:05.151028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.150989 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5648eb-b433-40b1-bc9b-b1d1f7106d81" containerName="payload-pre-processing" Nov 03 10:49:05.151028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151015 5086 cpu_manager.go:414] "containerMap: removing container" podUID="3fe0f981-a314-4e19-9d4b-cce9b3c08e6e" containerName="payload-processing" Nov 03 10:49:05.151028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151020 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe0f981-a314-4e19-9d4b-cce9b3c08e6e" containerName="payload-processing" Nov 03 10:49:05.151028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151028 5086 cpu_manager.go:414] "containerMap: removing container" podUID="be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" containerName="maas-api" Nov 03 10:49:05.151170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151033 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" containerName="maas-api" Nov 03 10:49:05.151170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151048 5086 cpu_manager.go:414] "containerMap: removing container" podUID="a4497595-63a7-4419-a909-569c8de96a6a" containerName="payload-processing" Nov 03 10:49:05.151170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151053 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4497595-63a7-4419-a909-569c8de96a6a" containerName="payload-processing" Nov 03 10:49:05.151170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151062 5086 cpu_manager.go:414] "containerMap: removing container" podUID="5877b26e-cbee-4c5f-adfc-0528c94acb92" containerName="payload-pre-processing" Nov 03 10:49:05.151170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151067 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5877b26e-cbee-4c5f-adfc-0528c94acb92" containerName="payload-pre-processing" Nov 03 10:49:05.151384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151172 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe0f981-a314-4e19-9d4b-cce9b3c08e6e" containerName="payload-processing" Nov 03 10:49:05.151384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151180 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5bd0bf-7131-4dc7-9d53-8acf5e1eab0d" containerName="maas-api" Nov 03 10:49:05.151384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151189 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4497595-63a7-4419-a909-569c8de96a6a" containerName="payload-processing" Nov 03 10:49:05.151384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151196 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="5877b26e-cbee-4c5f-adfc-0528c94acb92" containerName="payload-pre-processing" Nov 03 10:49:05.151384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.151206 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5648eb-b433-40b1-bc9b-b1d1f7106d81" containerName="payload-pre-processing" Nov 03 10:49:05.156565 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.156533 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:05.164144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.164117 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27"] Nov 03 10:49:05.166442 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.166415 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a51d55\"" Nov 03 10:49:05.166560 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.166543 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a51d55-dockercfg-rcnlf\"" Nov 03 10:49:05.171067 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.171045 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm"] Nov 03 10:49:05.176932 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.176907 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:05.200510 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.200480 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm"] Nov 03 10:49:05.300522 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.300486 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8l9t\" (UniqueName: \"kubernetes.io/projected/bc105700-2395-4afd-be20-56f83ac07a64-kube-api-access-x8l9t\") pod \"payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm\" (UID: \"bc105700-2395-4afd-be20-56f83ac07a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:05.300702 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.300531 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c04b53-cb5d-4fcf-a173-5682d67835ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27\" (UID: \"89c04b53-cb5d-4fcf-a173-5682d67835ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:05.300746 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.300703 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7c22\" (UniqueName: \"kubernetes.io/projected/89c04b53-cb5d-4fcf-a173-5682d67835ea-kube-api-access-l7c22\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27\" (UID: \"89c04b53-cb5d-4fcf-a173-5682d67835ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:05.300801 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.300788 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc105700-2395-4afd-be20-56f83ac07a64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm\" (UID: \"bc105700-2395-4afd-be20-56f83ac07a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:05.407366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.405993 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7c22\" (UniqueName: \"kubernetes.io/projected/89c04b53-cb5d-4fcf-a173-5682d67835ea-kube-api-access-l7c22\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27\" (UID: \"89c04b53-cb5d-4fcf-a173-5682d67835ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:05.407366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.406086 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc105700-2395-4afd-be20-56f83ac07a64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm\" (UID: \"bc105700-2395-4afd-be20-56f83ac07a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:05.407366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.406251 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8l9t\" (UniqueName: \"kubernetes.io/projected/bc105700-2395-4afd-be20-56f83ac07a64-kube-api-access-x8l9t\") pod \"payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm\" (UID: \"bc105700-2395-4afd-be20-56f83ac07a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:05.407366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.406280 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c04b53-cb5d-4fcf-a173-5682d67835ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27\" (UID: \"89c04b53-cb5d-4fcf-a173-5682d67835ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:05.407366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.407203 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c04b53-cb5d-4fcf-a173-5682d67835ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27\" (UID: \"89c04b53-cb5d-4fcf-a173-5682d67835ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:05.411313 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.408288 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc105700-2395-4afd-be20-56f83ac07a64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm\" (UID: \"bc105700-2395-4afd-be20-56f83ac07a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:05.430595 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.430555 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8l9t\" (UniqueName: \"kubernetes.io/projected/bc105700-2395-4afd-be20-56f83ac07a64-kube-api-access-x8l9t\") pod \"payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm\" (UID: \"bc105700-2395-4afd-be20-56f83ac07a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:05.431988 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.431960 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7c22\" (UniqueName: \"kubernetes.io/projected/89c04b53-cb5d-4fcf-a173-5682d67835ea-kube-api-access-l7c22\") pod \"payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27\" (UID: \"89c04b53-cb5d-4fcf-a173-5682d67835ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:05.475319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.475110 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:05.506387 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.506328 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:05.778117 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.778089 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27"] Nov 03 10:49:05.781030 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:05.780995 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c04b53_cb5d_4fcf_a173_5682d67835ea.slice/crio-d2ea8c0f0dccebf58747307a9aba839a61ff427b6b3e044d4457178aa41296dc WatchSource:0}: Error finding container d2ea8c0f0dccebf58747307a9aba839a61ff427b6b3e044d4457178aa41296dc: Status 404 returned error can't find the container with id d2ea8c0f0dccebf58747307a9aba839a61ff427b6b3e044d4457178aa41296dc Nov 03 10:49:05.830644 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:05.829954 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm"] Nov 03 10:49:05.830644 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:05.830103 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc105700_2395_4afd_be20_56f83ac07a64.slice/crio-bc35e2a6e8fb03fc49d11eefdefeee96aa4fba746fd8ce59481a098a2d73b595 WatchSource:0}: Error finding container bc35e2a6e8fb03fc49d11eefdefeee96aa4fba746fd8ce59481a098a2d73b595: Status 404 returned error can't find the container with id bc35e2a6e8fb03fc49d11eefdefeee96aa4fba746fd8ce59481a098a2d73b595 Nov 03 10:49:06.629463 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:06.629378 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" event={"ID":"bc105700-2395-4afd-be20-56f83ac07a64","Type":"ContainerStarted","Data":"bc35e2a6e8fb03fc49d11eefdefeee96aa4fba746fd8ce59481a098a2d73b595"} Nov 03 10:49:06.631000 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:06.630934 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" event={"ID":"89c04b53-cb5d-4fcf-a173-5682d67835ea","Type":"ContainerStarted","Data":"d2ea8c0f0dccebf58747307a9aba839a61ff427b6b3e044d4457178aa41296dc"} Nov 03 10:49:07.674451 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:07.674408 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" event={"ID":"bc105700-2395-4afd-be20-56f83ac07a64","Type":"ContainerStarted","Data":"85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d"} Nov 03 10:49:07.675517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:07.675488 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:07.678941 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:07.678894 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" event={"ID":"89c04b53-cb5d-4fcf-a173-5682d67835ea","Type":"ContainerStarted","Data":"e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a"} Nov 03 10:49:07.680619 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:07.680526 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:07.693945 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:07.693833 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" podStartSLOduration=1.087020385 podStartE2EDuration="2.693816233s" totalImagesPullingTime="1.606795848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:07.692764075 +0000 UTC m=+1225.677635406" watchObservedRunningTime="2026-11-03 10:49:07.693816233 +0000 UTC m=+1225.678687528" Nov 03 10:49:07.997284 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:07.997184 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" podStartSLOduration=1.2715673299999999 podStartE2EDuration="2.997168473s" totalImagesPullingTime="1.725601143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:07.71065399 +0000 UTC m=+1225.695525275" watchObservedRunningTime="2026-11-03 10:49:07.997168473 +0000 UTC m=+1225.982039802" Nov 03 10:49:08.000576 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.000549 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq"] Nov 03 10:49:08.006049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.006019 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:08.009942 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.009638 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-de30a0\"" Nov 03 10:49:08.009942 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.009837 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-de30a0-dockercfg-hl2xh\"" Nov 03 10:49:08.014045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.014022 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq"] Nov 03 10:49:08.036045 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.035993 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd"] Nov 03 10:49:08.045578 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.045551 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd"] Nov 03 10:49:08.045743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.045655 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:08.216275 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.216234 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw9vt\" (UniqueName: \"kubernetes.io/projected/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-kube-api-access-cw9vt\") pod \"payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd\" (UID: \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:08.250515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.250432 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzj8z\" (UniqueName: \"kubernetes.io/projected/b352759f-5a51-4019-bf52-99f1635f80f6-kube-api-access-rzj8z\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq\" (UID: \"b352759f-5a51-4019-bf52-99f1635f80f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:08.250904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.250880 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b352759f-5a51-4019-bf52-99f1635f80f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq\" (UID: \"b352759f-5a51-4019-bf52-99f1635f80f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:08.251060 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.251042 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd\" (UID: \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:08.356316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.352334 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw9vt\" (UniqueName: \"kubernetes.io/projected/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-kube-api-access-cw9vt\") pod \"payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd\" (UID: \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:08.356316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.352484 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzj8z\" (UniqueName: \"kubernetes.io/projected/b352759f-5a51-4019-bf52-99f1635f80f6-kube-api-access-rzj8z\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq\" (UID: \"b352759f-5a51-4019-bf52-99f1635f80f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:08.356316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.352832 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b352759f-5a51-4019-bf52-99f1635f80f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq\" (UID: \"b352759f-5a51-4019-bf52-99f1635f80f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:08.356316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.352881 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd\" (UID: \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:08.356316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.353855 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd\" (UID: \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:08.360199 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.358852 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b352759f-5a51-4019-bf52-99f1635f80f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq\" (UID: \"b352759f-5a51-4019-bf52-99f1635f80f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:08.365043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.365017 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw9vt\" (UniqueName: \"kubernetes.io/projected/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-kube-api-access-cw9vt\") pod \"payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd\" (UID: \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:08.366234 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.366017 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:08.367065 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.367044 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzj8z\" (UniqueName: \"kubernetes.io/projected/b352759f-5a51-4019-bf52-99f1635f80f6-kube-api-access-rzj8z\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq\" (UID: \"b352759f-5a51-4019-bf52-99f1635f80f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:08.626630 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.626584 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:08.664681 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.656113 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd"] Nov 03 10:49:08.737150 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.737123 5086 generic.go:356] "Generic (PLEG): container finished" podID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerID="9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756" exitCode=1 Nov 03 10:49:08.737960 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.737937 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" event={"ID":"ef6d124d-4f64-41fc-ad08-a3878a62da29","Type":"ContainerDied","Data":"9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756"} Nov 03 10:49:08.740564 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.738399 5086 scope.go:122] "RemoveContainer" containerID="9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756" Nov 03 10:49:08.987385 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:08.987355 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb352759f_5a51_4019_bf52_99f1635f80f6.slice/crio-da7454cfd66de1c71b58a9ef5a3e76d0c92bd79db2d8b4ff95b2f2a2c7a33901 WatchSource:0}: Error finding container da7454cfd66de1c71b58a9ef5a3e76d0c92bd79db2d8b4ff95b2f2a2c7a33901: Status 404 returned error can't find the container with id da7454cfd66de1c71b58a9ef5a3e76d0c92bd79db2d8b4ff95b2f2a2c7a33901 Nov 03 10:49:08.988973 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:08.988948 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq"] Nov 03 10:49:09.746108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:09.746072 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" event={"ID":"b352759f-5a51-4019-bf52-99f1635f80f6","Type":"ContainerStarted","Data":"55e2811e36c428b224c7857c55209c743215feb14228757cd7804444655332ae"} Nov 03 10:49:09.746606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:09.746311 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:09.746606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:09.746353 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" event={"ID":"b352759f-5a51-4019-bf52-99f1635f80f6","Type":"ContainerStarted","Data":"da7454cfd66de1c71b58a9ef5a3e76d0c92bd79db2d8b4ff95b2f2a2c7a33901"} Nov 03 10:49:09.747862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:09.747830 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" event={"ID":"92a8eb5c-0647-41ba-a265-cd61bb6bd77d","Type":"ContainerStarted","Data":"2e7c9873886aa2a1ff8bf0063a7a5cf0d3e8e172c2b0c22b17f4f4b1c4e67c7a"} Nov 03 10:49:09.747992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:09.747886 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:09.747992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:09.747908 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" event={"ID":"92a8eb5c-0647-41ba-a265-cd61bb6bd77d","Type":"ContainerStarted","Data":"d4708b978a3d20c2de3eb936b3a0a75a7c156bc544e2b0a1164334cbabaa2608"} Nov 03 10:49:09.749892 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:09.749863 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" event={"ID":"ef6d124d-4f64-41fc-ad08-a3878a62da29","Type":"ContainerStarted","Data":"1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792"} Nov 03 10:49:09.763521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:09.763473 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" podStartSLOduration=2.763461779 podStartE2EDuration="2.763461779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:09.760500309 +0000 UTC m=+1227.745371594" watchObservedRunningTime="2026-11-03 10:49:09.763461779 +0000 UTC m=+1227.748333064" Nov 03 10:49:09.790938 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:09.790889 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" podStartSLOduration=1.790875386 podStartE2EDuration="1.790875386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:09.788583796 +0000 UTC m=+1227.773455072" watchObservedRunningTime="2026-11-03 10:49:09.790875386 +0000 UTC m=+1227.775746688" Nov 03 10:49:13.051558 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.051526 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc"] Nov 03 10:49:13.057283 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.057260 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.074508 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.067553 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-de30a0-serving-cert\"" Nov 03 10:49:13.081366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.076804 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc"] Nov 03 10:49:13.179371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.179331 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-metrics-certs\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.179532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.179459 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.179532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.179515 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e1019a-4d07-44ed-935e-67d588004dd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.179663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.179547 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpfd\" (UniqueName: \"kubernetes.io/projected/d1e1019a-4d07-44ed-935e-67d588004dd1-kube-api-access-xlpfd\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.291525 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.291193 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-metrics-certs\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.291525 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.291325 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.291525 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.291396 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e1019a-4d07-44ed-935e-67d588004dd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.291525 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.291455 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpfd\" (UniqueName: \"kubernetes.io/projected/d1e1019a-4d07-44ed-935e-67d588004dd1-kube-api-access-xlpfd\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.325319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.319858 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e1019a-4d07-44ed-935e-67d588004dd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.325319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.321001 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-metrics-certs\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.325319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.323570 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.336099 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.336018 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpfd\" (UniqueName: \"kubernetes.io/projected/d1e1019a-4d07-44ed-935e-67d588004dd1-kube-api-access-xlpfd\") pod \"maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.395830 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.395528 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:13.719563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.718272 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc"] Nov 03 10:49:13.719563 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:13.719492 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e1019a_4d07_44ed_935e_67d588004dd1.slice/crio-33ecd89c0aa1443a7c60714e7ad00d5883cd6e0583981126aacb4031d4f0e55d WatchSource:0}: Error finding container 33ecd89c0aa1443a7c60714e7ad00d5883cd6e0583981126aacb4031d4f0e55d: Status 404 returned error can't find the container with id 33ecd89c0aa1443a7c60714e7ad00d5883cd6e0583981126aacb4031d4f0e55d Nov 03 10:49:13.780261 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:13.780228 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" event={"ID":"d1e1019a-4d07-44ed-935e-67d588004dd1","Type":"ContainerStarted","Data":"33ecd89c0aa1443a7c60714e7ad00d5883cd6e0583981126aacb4031d4f0e55d"} Nov 03 10:49:14.750880 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:14.750841 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:49:14.756790 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:14.756761 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:49:14.791987 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:14.791939 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" event={"ID":"d1e1019a-4d07-44ed-935e-67d588004dd1","Type":"ContainerStarted","Data":"1230d4ad195318f3834c152b96a32386dd9294d591a2f91996cffb3f7a25f507"} Nov 03 10:49:14.792165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:14.792027 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:14.823936 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:14.823869 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" podStartSLOduration=1.3981962540000001 podStartE2EDuration="1.823848394s" totalImagesPullingTime="425.65214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:14.815669315 +0000 UTC m=+1232.800540600" watchObservedRunningTime="2026-11-03 10:49:14.823848394 +0000 UTC m=+1232.808719678" Nov 03 10:49:19.752163 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:19.752126 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:49:19.752652 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:19.752219 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:49:19.985665 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:19.985627 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn"] Nov 03 10:49:19.990872 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:19.990843 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:19.994893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:19.994863 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4d9549-dockercfg-r6ph6\"" Nov 03 10:49:19.999120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:19.999086 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4d9549\"" Nov 03 10:49:19.999532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:19.999506 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn"] Nov 03 10:49:20.040516 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.040482 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hpvj\" (UniqueName: \"kubernetes.io/projected/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-kube-api-access-8hpvj\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn\" (UID: \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:20.040784 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.040760 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn\" (UID: \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:20.050124 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.050088 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl"] Nov 03 10:49:20.054997 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.054977 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:20.064571 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.064540 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl"] Nov 03 10:49:20.162365 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.152290 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hpvj\" (UniqueName: \"kubernetes.io/projected/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-kube-api-access-8hpvj\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn\" (UID: \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:20.162365 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.152556 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn\" (UID: \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:20.170774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.170724 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn\" (UID: \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:20.228166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.228127 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hpvj\" (UniqueName: \"kubernetes.io/projected/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-kube-api-access-8hpvj\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn\" (UID: \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:20.254691 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.254593 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43b6405-ac5b-46af-992c-c021aeb9a908-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl\" (UID: \"b43b6405-ac5b-46af-992c-c021aeb9a908\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:20.254869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.254731 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pw5n\" (UniqueName: \"kubernetes.io/projected/b43b6405-ac5b-46af-992c-c021aeb9a908-kube-api-access-4pw5n\") pod \"payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl\" (UID: \"b43b6405-ac5b-46af-992c-c021aeb9a908\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:20.317916 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.317883 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:20.357625 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.357590 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43b6405-ac5b-46af-992c-c021aeb9a908-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl\" (UID: \"b43b6405-ac5b-46af-992c-c021aeb9a908\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:20.358073 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.357717 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pw5n\" (UniqueName: \"kubernetes.io/projected/b43b6405-ac5b-46af-992c-c021aeb9a908-kube-api-access-4pw5n\") pod \"payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl\" (UID: \"b43b6405-ac5b-46af-992c-c021aeb9a908\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:20.358833 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.358802 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43b6405-ac5b-46af-992c-c021aeb9a908-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl\" (UID: \"b43b6405-ac5b-46af-992c-c021aeb9a908\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:20.370468 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.369818 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pw5n\" (UniqueName: \"kubernetes.io/projected/b43b6405-ac5b-46af-992c-c021aeb9a908-kube-api-access-4pw5n\") pod \"payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl\" (UID: \"b43b6405-ac5b-46af-992c-c021aeb9a908\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:20.384621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.384589 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:20.641137 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.641106 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn"] Nov 03 10:49:20.642031 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:20.642003 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b75cf94_7724_4b8f_b42c_42aa91b25bd4.slice/crio-7f880f69beb01d752d2f142236432e7e2b333519ebebe14ef8a33638bd9de169 WatchSource:0}: Error finding container 7f880f69beb01d752d2f142236432e7e2b333519ebebe14ef8a33638bd9de169: Status 404 returned error can't find the container with id 7f880f69beb01d752d2f142236432e7e2b333519ebebe14ef8a33638bd9de169 Nov 03 10:49:20.713206 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:20.713172 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43b6405_ac5b_46af_992c_c021aeb9a908.slice/crio-f6b3d975de8f12f66eaebb3e2ce41fa82f2c6dfacb74bdb151963d58f50c0d8a WatchSource:0}: Error finding container f6b3d975de8f12f66eaebb3e2ce41fa82f2c6dfacb74bdb151963d58f50c0d8a: Status 404 returned error can't find the container with id f6b3d975de8f12f66eaebb3e2ce41fa82f2c6dfacb74bdb151963d58f50c0d8a Nov 03 10:49:20.716548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.715597 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl"] Nov 03 10:49:20.761474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.761451 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:20.761786 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.761532 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:20.811587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.811554 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:49:20.934886 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.934788 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" event={"ID":"9b75cf94-7724-4b8f-b42c-42aa91b25bd4","Type":"ContainerStarted","Data":"201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a"} Nov 03 10:49:20.934886 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.934884 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" event={"ID":"9b75cf94-7724-4b8f-b42c-42aa91b25bd4","Type":"ContainerStarted","Data":"7f880f69beb01d752d2f142236432e7e2b333519ebebe14ef8a33638bd9de169"} Nov 03 10:49:20.935123 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.934990 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:20.937199 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.937171 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" event={"ID":"b43b6405-ac5b-46af-992c-c021aeb9a908","Type":"ContainerStarted","Data":"aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d"} Nov 03 10:49:20.937340 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.937216 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" event={"ID":"b43b6405-ac5b-46af-992c-c021aeb9a908","Type":"ContainerStarted","Data":"f6b3d975de8f12f66eaebb3e2ce41fa82f2c6dfacb74bdb151963d58f50c0d8a"} Nov 03 10:49:20.937426 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.937347 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:20.955366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.955319 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" podStartSLOduration=1.955306056 podStartE2EDuration="1.955306056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:20.951957442 +0000 UTC m=+1238.936828731" watchObservedRunningTime="2026-11-03 10:49:20.955306056 +0000 UTC m=+1238.940177330" Nov 03 10:49:20.971657 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:20.971616 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" podStartSLOduration=0.971603973 podStartE2EDuration="971.603973ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:20.96766743 +0000 UTC m=+1238.952538707" watchObservedRunningTime="2026-11-03 10:49:20.971603973 +0000 UTC m=+1238.956475256" Nov 03 10:49:24.281976 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:24.281932 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:49:26.748339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:26.748291 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd"] Nov 03 10:49:26.764216 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:26.763068 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq"] Nov 03 10:49:28.868689 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:28.868656 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf"] Nov 03 10:49:28.885136 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:28.885105 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:28.887099 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:28.887075 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf"] Nov 03 10:49:28.889021 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:28.889000 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4d9549-serving-cert\"" Nov 03 10:49:29.016923 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.016890 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8bth\" (UniqueName: \"kubernetes.io/projected/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-kube-api-access-w8bth\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.017112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.017015 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.017112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.017058 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.017241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.017199 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-metrics-certs\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.159693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.134541 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.159693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.134757 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-metrics-certs\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.159693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.135084 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8bth\" (UniqueName: \"kubernetes.io/projected/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-kube-api-access-w8bth\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.159693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.135232 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.159693 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.158967 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.170170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.170050 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.173163 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.171909 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8bth\" (UniqueName: \"kubernetes.io/projected/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-kube-api-access-w8bth\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.177760 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.177714 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-metrics-certs\") pod \"maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.207577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.207545 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:29.521457 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:29.509771 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f32e25_d4ec_4b3e_8fa0_0cd14b21b916.slice/crio-09d49a603c8a35b1e965bdb6f1f991ed0069bd2a75f8d95a698ef54ef8c325fd WatchSource:0}: Error finding container 09d49a603c8a35b1e965bdb6f1f991ed0069bd2a75f8d95a698ef54ef8c325fd: Status 404 returned error can't find the container with id 09d49a603c8a35b1e965bdb6f1f991ed0069bd2a75f8d95a698ef54ef8c325fd Nov 03 10:49:29.521457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:29.513684 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf"] Nov 03 10:49:30.022126 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.022093 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" event={"ID":"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916","Type":"ContainerStarted","Data":"09d49a603c8a35b1e965bdb6f1f991ed0069bd2a75f8d95a698ef54ef8c325fd"} Nov 03 10:49:30.719036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.718997 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd"] Nov 03 10:49:30.725937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.725893 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:30.731629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.731600 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aa420c-dockercfg-xb9wm\"" Nov 03 10:49:30.732212 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.732182 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd"] Nov 03 10:49:30.808089 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.808037 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa420c\"" Nov 03 10:49:30.897748 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.893723 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd\" (UID: \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:30.897748 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.893884 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dksw7\" (UniqueName: \"kubernetes.io/projected/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-kube-api-access-dksw7\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd\" (UID: \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:30.995168 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.995077 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd\" (UID: \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:30.995366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.995219 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dksw7\" (UniqueName: \"kubernetes.io/projected/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-kube-api-access-dksw7\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd\" (UID: \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:30.995934 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:30.995908 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd\" (UID: \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:31.003871 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.003843 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dksw7\" (UniqueName: \"kubernetes.io/projected/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-kube-api-access-dksw7\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd\" (UID: \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:31.032081 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.032034 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" event={"ID":"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916","Type":"ContainerStarted","Data":"9b4314073d3fe909bcf9f8420ad9610a8ffdad5f2ca4e08d13af8731fed5b2a8"} Nov 03 10:49:31.032466 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.032230 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:31.049105 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.048481 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:31.050064 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.050011 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" podStartSLOduration=2.592647695 podStartE2EDuration="3.049996114s" totalImagesPullingTime="457.348419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:31.048527747 +0000 UTC m=+1249.033399031" watchObservedRunningTime="2026-11-03 10:49:31.049996114 +0000 UTC m=+1249.034867397" Nov 03 10:49:31.068805 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.068776 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582"] Nov 03 10:49:31.087875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.087842 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k"] Nov 03 10:49:31.378743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.378700 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd"] Nov 03 10:49:31.408976 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:31.408920 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b05c6e5_6d6b_4f96_84d0_3d85a35efb67.slice/crio-0f0e9fe20ded38f5eba8343c2b5f443258f6240e697cf7f987c781fcb9d61dfb WatchSource:0}: Error finding container 0f0e9fe20ded38f5eba8343c2b5f443258f6240e697cf7f987c781fcb9d61dfb: Status 404 returned error can't find the container with id 0f0e9fe20ded38f5eba8343c2b5f443258f6240e697cf7f987c781fcb9d61dfb Nov 03 10:49:31.749396 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.749277 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" podUID="92a8eb5c-0647-41ba-a265-cd61bb6bd77d" containerName="payload-processing" containerID="cri-o://2e7c9873886aa2a1ff8bf0063a7a5cf0d3e8e172c2b0c22b17f4f4b1c4e67c7a" gracePeriod=25 Nov 03 10:49:31.764893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.764853 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" podUID="b352759f-5a51-4019-bf52-99f1635f80f6" containerName="payload-pre-processing" containerID="cri-o://55e2811e36c428b224c7857c55209c743215feb14228757cd7804444655332ae" gracePeriod=25 Nov 03 10:49:31.950238 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.950169 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:31.952037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:31.951414 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:32.046535 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.046495 5086 generic.go:356] "Generic (PLEG): container finished" podID="b352759f-5a51-4019-bf52-99f1635f80f6" containerID="55e2811e36c428b224c7857c55209c743215feb14228757cd7804444655332ae" exitCode=0 Nov 03 10:49:32.047208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.047046 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" event={"ID":"b352759f-5a51-4019-bf52-99f1635f80f6","Type":"ContainerDied","Data":"55e2811e36c428b224c7857c55209c743215feb14228757cd7804444655332ae"} Nov 03 10:49:32.050670 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.050066 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" event={"ID":"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67","Type":"ContainerStarted","Data":"c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535"} Nov 03 10:49:32.050670 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.050127 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:32.050670 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.050176 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" event={"ID":"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67","Type":"ContainerStarted","Data":"0f0e9fe20ded38f5eba8343c2b5f443258f6240e697cf7f987c781fcb9d61dfb"} Nov 03 10:49:32.053833 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.053808 5086 generic.go:356] "Generic (PLEG): container finished" podID="92a8eb5c-0647-41ba-a265-cd61bb6bd77d" containerID="2e7c9873886aa2a1ff8bf0063a7a5cf0d3e8e172c2b0c22b17f4f4b1c4e67c7a" exitCode=0 Nov 03 10:49:32.054282 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.054253 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" event={"ID":"92a8eb5c-0647-41ba-a265-cd61bb6bd77d","Type":"ContainerDied","Data":"2e7c9873886aa2a1ff8bf0063a7a5cf0d3e8e172c2b0c22b17f4f4b1c4e67c7a"} Nov 03 10:49:32.089250 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.089220 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:32.122136 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.122067 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" podStartSLOduration=2.122052017 podStartE2EDuration="2.122052017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:32.074870791 +0000 UTC m=+1250.059742075" watchObservedRunningTime="2026-11-03 10:49:32.122052017 +0000 UTC m=+1250.106923299" Nov 03 10:49:32.143652 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.143625 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:32.226695 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.226654 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-kube-api-access-cw9vt\" (UniqueName: \"kubernetes.io/projected/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-kube-api-access-cw9vt\") pod \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\" (UID: \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\") " Nov 03 10:49:32.227280 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.227258 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b352759f-5a51-4019-bf52-99f1635f80f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b352759f-5a51-4019-bf52-99f1635f80f6-plugins-config-volume\") pod \"b352759f-5a51-4019-bf52-99f1635f80f6\" (UID: \"b352759f-5a51-4019-bf52-99f1635f80f6\") " Nov 03 10:49:32.227817 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.227794 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-plugins-config-volume\") pod \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\" (UID: \"92a8eb5c-0647-41ba-a265-cd61bb6bd77d\") " Nov 03 10:49:32.228382 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.227943 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b352759f-5a51-4019-bf52-99f1635f80f6-plugins-config-volume" pod "b352759f-5a51-4019-bf52-99f1635f80f6" (UID: "b352759f-5a51-4019-bf52-99f1635f80f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:32.232778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.228234 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-plugins-config-volume" pod "92a8eb5c-0647-41ba-a265-cd61bb6bd77d" (UID: "92a8eb5c-0647-41ba-a265-cd61bb6bd77d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:32.232778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.228874 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b352759f-5a51-4019-bf52-99f1635f80f6-kube-api-access-rzj8z\" (UniqueName: \"kubernetes.io/projected/b352759f-5a51-4019-bf52-99f1635f80f6-kube-api-access-rzj8z\") pod \"b352759f-5a51-4019-bf52-99f1635f80f6\" (UID: \"b352759f-5a51-4019-bf52-99f1635f80f6\") " Nov 03 10:49:32.232778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.232227 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:32.232778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.232249 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b352759f-5a51-4019-bf52-99f1635f80f6-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:32.233684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.233657 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-kube-api-access-cw9vt" pod "92a8eb5c-0647-41ba-a265-cd61bb6bd77d" (UID: "92a8eb5c-0647-41ba-a265-cd61bb6bd77d"). InnerVolumeSpecName "kube-api-access-cw9vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:32.233784 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.233704 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b352759f-5a51-4019-bf52-99f1635f80f6-kube-api-access-rzj8z" pod "b352759f-5a51-4019-bf52-99f1635f80f6" (UID: "b352759f-5a51-4019-bf52-99f1635f80f6"). InnerVolumeSpecName "kube-api-access-rzj8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:32.333683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.333644 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzj8z\" (UniqueName: \"kubernetes.io/projected/b352759f-5a51-4019-bf52-99f1635f80f6-kube-api-access-rzj8z\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:32.333683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:32.333680 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw9vt\" (UniqueName: \"kubernetes.io/projected/92a8eb5c-0647-41ba-a265-cd61bb6bd77d-kube-api-access-cw9vt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:33.063018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.062986 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" Nov 03 10:49:33.063018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.063004 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq" event={"ID":"b352759f-5a51-4019-bf52-99f1635f80f6","Type":"ContainerDied","Data":"da7454cfd66de1c71b58a9ef5a3e76d0c92bd79db2d8b4ff95b2f2a2c7a33901"} Nov 03 10:49:33.063625 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.063073 5086 scope.go:122] "RemoveContainer" containerID="55e2811e36c428b224c7857c55209c743215feb14228757cd7804444655332ae" Nov 03 10:49:33.064888 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.064865 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" Nov 03 10:49:33.064995 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.064904 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd" event={"ID":"92a8eb5c-0647-41ba-a265-cd61bb6bd77d","Type":"ContainerDied","Data":"d4708b978a3d20c2de3eb936b3a0a75a7c156bc544e2b0a1164334cbabaa2608"} Nov 03 10:49:33.088925 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.088902 5086 scope.go:122] "RemoveContainer" containerID="2e7c9873886aa2a1ff8bf0063a7a5cf0d3e8e172c2b0c22b17f4f4b1c4e67c7a" Nov 03 10:49:33.094953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.094917 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq"] Nov 03 10:49:33.099793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.099767 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-774d9fb5ff-5vxjq"] Nov 03 10:49:33.110119 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.110095 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd"] Nov 03 10:49:33.113784 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:33.113762 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-555947989f-d77qd"] Nov 03 10:49:34.172051 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:34.172010 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a8eb5c-0647-41ba-a265-cd61bb6bd77d" path="/var/lib/kubelet/pods/92a8eb5c-0647-41ba-a265-cd61bb6bd77d/volumes" Nov 03 10:49:34.172545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:34.172527 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b352759f-5a51-4019-bf52-99f1635f80f6" path="/var/lib/kubelet/pods/b352759f-5a51-4019-bf52-99f1635f80f6/volumes" Nov 03 10:49:36.069897 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.069855 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" podUID="c3a0714d-13e6-4863-a6f2-79e7e883e73f" containerName="payload-processing" containerID="cri-o://c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a" gracePeriod=25 Nov 03 10:49:36.088958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.088920 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" podUID="c404f7f7-80ab-418e-acae-f76716ca7b16" containerName="payload-pre-processing" containerID="cri-o://d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519" gracePeriod=25 Nov 03 10:49:36.570685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.570641 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:49:36.664834 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.664767 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:49:36.708835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.708615 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a0714d-13e6-4863-a6f2-79e7e883e73f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a0714d-13e6-4863-a6f2-79e7e883e73f-plugins-config-volume\") pod \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\" (UID: \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\") " Nov 03 10:49:36.708835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.708714 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c404f7f7-80ab-418e-acae-f76716ca7b16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c404f7f7-80ab-418e-acae-f76716ca7b16-plugins-config-volume\") pod \"c404f7f7-80ab-418e-acae-f76716ca7b16\" (UID: \"c404f7f7-80ab-418e-acae-f76716ca7b16\") " Nov 03 10:49:36.709276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.709150 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c404f7f7-80ab-418e-acae-f76716ca7b16-kube-api-access-tx6fd\" (UniqueName: \"kubernetes.io/projected/c404f7f7-80ab-418e-acae-f76716ca7b16-kube-api-access-tx6fd\") pod \"c404f7f7-80ab-418e-acae-f76716ca7b16\" (UID: \"c404f7f7-80ab-418e-acae-f76716ca7b16\") " Nov 03 10:49:36.709413 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.709378 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a0714d-13e6-4863-a6f2-79e7e883e73f-kube-api-access-zq7lq\" (UniqueName: \"kubernetes.io/projected/c3a0714d-13e6-4863-a6f2-79e7e883e73f-kube-api-access-zq7lq\") pod \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\" (UID: \"c3a0714d-13e6-4863-a6f2-79e7e883e73f\") " Nov 03 10:49:36.709895 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.709870 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a0714d-13e6-4863-a6f2-79e7e883e73f-plugins-config-volume" pod "c3a0714d-13e6-4863-a6f2-79e7e883e73f" (UID: "c3a0714d-13e6-4863-a6f2-79e7e883e73f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:36.710252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.710232 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a0714d-13e6-4863-a6f2-79e7e883e73f-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:36.710534 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.710513 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c404f7f7-80ab-418e-acae-f76716ca7b16-plugins-config-volume" pod "c404f7f7-80ab-418e-acae-f76716ca7b16" (UID: "c404f7f7-80ab-418e-acae-f76716ca7b16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:36.712932 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.712910 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c404f7f7-80ab-418e-acae-f76716ca7b16-kube-api-access-tx6fd" pod "c404f7f7-80ab-418e-acae-f76716ca7b16" (UID: "c404f7f7-80ab-418e-acae-f76716ca7b16"). InnerVolumeSpecName "kube-api-access-tx6fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:36.713136 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.713117 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a0714d-13e6-4863-a6f2-79e7e883e73f-kube-api-access-zq7lq" pod "c3a0714d-13e6-4863-a6f2-79e7e883e73f" (UID: "c3a0714d-13e6-4863-a6f2-79e7e883e73f"). InnerVolumeSpecName "kube-api-access-zq7lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:36.812106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.812062 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c404f7f7-80ab-418e-acae-f76716ca7b16-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:36.812106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.812100 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx6fd\" (UniqueName: \"kubernetes.io/projected/c404f7f7-80ab-418e-acae-f76716ca7b16-kube-api-access-tx6fd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:36.812106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:36.812114 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq7lq\" (UniqueName: \"kubernetes.io/projected/c3a0714d-13e6-4863-a6f2-79e7e883e73f-kube-api-access-zq7lq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:37.060511 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.060439 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:49:37.131696 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.131656 5086 generic.go:356] "Generic (PLEG): container finished" podID="c404f7f7-80ab-418e-acae-f76716ca7b16" containerID="d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519" exitCode=0 Nov 03 10:49:37.132173 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.131940 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" event={"ID":"c404f7f7-80ab-418e-acae-f76716ca7b16","Type":"ContainerDied","Data":"d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519"} Nov 03 10:49:37.132173 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.131993 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" event={"ID":"c404f7f7-80ab-418e-acae-f76716ca7b16","Type":"ContainerDied","Data":"4353c3a6a7f382a2cec6b1d3e9b41aecffacaa1557fc5a985fc8bd806c0f8722"} Nov 03 10:49:37.132173 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.132026 5086 scope.go:122] "RemoveContainer" containerID="d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519" Nov 03 10:49:37.132173 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.132139 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k" Nov 03 10:49:37.209598 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.206031 5086 generic.go:356] "Generic (PLEG): container finished" podID="c3a0714d-13e6-4863-a6f2-79e7e883e73f" containerID="c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a" exitCode=0 Nov 03 10:49:37.209598 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.206152 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" event={"ID":"c3a0714d-13e6-4863-a6f2-79e7e883e73f","Type":"ContainerDied","Data":"c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a"} Nov 03 10:49:37.209598 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.206206 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" event={"ID":"c3a0714d-13e6-4863-a6f2-79e7e883e73f","Type":"ContainerDied","Data":"ce90212620e5fade3a331a6ece7ad2468142624bdd1e790d33fbce1d88386867"} Nov 03 10:49:37.209598 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.206290 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582" Nov 03 10:49:37.262779 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.262753 5086 scope.go:122] "RemoveContainer" containerID="d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519" Nov 03 10:49:37.263179 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:37.263157 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519\": container with ID starting with d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519 not found: ID does not exist" containerID="d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519" Nov 03 10:49:37.263339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.263190 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519"} err="failed to get container status \"d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519\": rpc error: code = NotFound desc = could not find container \"d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519\": container with ID starting with d0c9826321cfbff64e985bdd6a54e317d5b64c54bad59a20efcf1be4626fc519 not found: ID does not exist" Nov 03 10:49:37.263339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.263215 5086 scope.go:122] "RemoveContainer" containerID="c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a" Nov 03 10:49:37.294510 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.294480 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582"] Nov 03 10:49:37.299958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.299925 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-90bad4-5f94bdb77f-w6582"] Nov 03 10:49:37.303478 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.303154 5086 scope.go:122] "RemoveContainer" containerID="c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a" Nov 03 10:49:37.305383 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:37.305348 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a\": container with ID starting with c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a not found: ID does not exist" containerID="c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a" Nov 03 10:49:37.305507 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.305394 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a"} err="failed to get container status \"c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a\": rpc error: code = NotFound desc = could not find container \"c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a\": container with ID starting with c9a5a258dbbfc59c307918d0dd76163fa4d14c631a500c2bc135ace93b81f40a not found: ID does not exist" Nov 03 10:49:37.324095 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.324037 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k"] Nov 03 10:49:37.332339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:37.332272 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-90bad4-56b44f59d7-vl44k"] Nov 03 10:49:38.172401 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.172363 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a0714d-13e6-4863-a6f2-79e7e883e73f" path="/var/lib/kubelet/pods/c3a0714d-13e6-4863-a6f2-79e7e883e73f/volumes" Nov 03 10:49:38.172937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.172915 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c404f7f7-80ab-418e-acae-f76716ca7b16" path="/var/lib/kubelet/pods/c404f7f7-80ab-418e-acae-f76716ca7b16/volumes" Nov 03 10:49:38.193253 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.193219 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw"] Nov 03 10:49:38.194454 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194433 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c404f7f7-80ab-418e-acae-f76716ca7b16" containerName="payload-pre-processing" Nov 03 10:49:38.194519 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194456 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c404f7f7-80ab-418e-acae-f76716ca7b16" containerName="payload-pre-processing" Nov 03 10:49:38.194519 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194504 5086 cpu_manager.go:414] "containerMap: removing container" podUID="92a8eb5c-0647-41ba-a265-cd61bb6bd77d" containerName="payload-processing" Nov 03 10:49:38.194519 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194512 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a8eb5c-0647-41ba-a265-cd61bb6bd77d" containerName="payload-processing" Nov 03 10:49:38.194615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194529 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c3a0714d-13e6-4863-a6f2-79e7e883e73f" containerName="payload-processing" Nov 03 10:49:38.194615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194535 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a0714d-13e6-4863-a6f2-79e7e883e73f" containerName="payload-processing" Nov 03 10:49:38.194615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194562 5086 cpu_manager.go:414] "containerMap: removing container" podUID="b352759f-5a51-4019-bf52-99f1635f80f6" containerName="payload-pre-processing" Nov 03 10:49:38.194615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194569 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b352759f-5a51-4019-bf52-99f1635f80f6" containerName="payload-pre-processing" Nov 03 10:49:38.194760 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194747 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a0714d-13e6-4863-a6f2-79e7e883e73f" containerName="payload-processing" Nov 03 10:49:38.194798 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194772 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="b352759f-5a51-4019-bf52-99f1635f80f6" containerName="payload-pre-processing" Nov 03 10:49:38.194798 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194787 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c404f7f7-80ab-418e-acae-f76716ca7b16" containerName="payload-pre-processing" Nov 03 10:49:38.194859 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.194804 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a8eb5c-0647-41ba-a265-cd61bb6bd77d" containerName="payload-processing" Nov 03 10:49:38.202147 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.202104 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:38.205466 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.205434 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw"] Nov 03 10:49:38.225659 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.225622 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-de30a0\"" Nov 03 10:49:38.357356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.356224 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046018fb-6293-435f-a62e-9d1fccde20b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw\" (UID: \"046018fb-6293-435f-a62e-9d1fccde20b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:38.357356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.356289 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzk9\" (UniqueName: \"kubernetes.io/projected/046018fb-6293-435f-a62e-9d1fccde20b3-kube-api-access-rrzk9\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw\" (UID: \"046018fb-6293-435f-a62e-9d1fccde20b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:38.405075 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.405038 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls"] Nov 03 10:49:38.415787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.415757 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls"] Nov 03 10:49:38.416118 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.416100 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:38.463397 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.454700 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-de30a0-dockercfg-mndqp\"" Nov 03 10:49:38.531625 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.531563 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046018fb-6293-435f-a62e-9d1fccde20b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw\" (UID: \"046018fb-6293-435f-a62e-9d1fccde20b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:38.531997 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.531957 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzk9\" (UniqueName: \"kubernetes.io/projected/046018fb-6293-435f-a62e-9d1fccde20b3-kube-api-access-rrzk9\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw\" (UID: \"046018fb-6293-435f-a62e-9d1fccde20b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:38.533608 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.533573 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046018fb-6293-435f-a62e-9d1fccde20b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw\" (UID: \"046018fb-6293-435f-a62e-9d1fccde20b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:38.540262 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.540239 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzk9\" (UniqueName: \"kubernetes.io/projected/046018fb-6293-435f-a62e-9d1fccde20b3-kube-api-access-rrzk9\") pod \"payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw\" (UID: \"046018fb-6293-435f-a62e-9d1fccde20b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:38.546059 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.546038 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:38.635761 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.635716 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebe5575-ee5c-448e-8858-0c6e3803efe1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls\" (UID: \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:38.636019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.635986 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txdc\" (UniqueName: \"kubernetes.io/projected/3ebe5575-ee5c-448e-8858-0c6e3803efe1-kube-api-access-6txdc\") pod \"payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls\" (UID: \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:38.737407 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.737324 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebe5575-ee5c-448e-8858-0c6e3803efe1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls\" (UID: \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:38.737557 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.737519 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txdc\" (UniqueName: \"kubernetes.io/projected/3ebe5575-ee5c-448e-8858-0c6e3803efe1-kube-api-access-6txdc\") pod \"payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls\" (UID: \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:38.738122 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.738102 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebe5575-ee5c-448e-8858-0c6e3803efe1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls\" (UID: \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:38.745463 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.745435 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txdc\" (UniqueName: \"kubernetes.io/projected/3ebe5575-ee5c-448e-8858-0c6e3803efe1-kube-api-access-6txdc\") pod \"payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls\" (UID: \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:38.836192 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.836160 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:38.836377 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:38.836353 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046018fb_6293_435f_a62e_9d1fccde20b3.slice/crio-a1d2f7ab1ee7f919615891e0dbee5b6722050acfbab408e8ca06450d54ca608b WatchSource:0}: Error finding container a1d2f7ab1ee7f919615891e0dbee5b6722050acfbab408e8ca06450d54ca608b: Status 404 returned error can't find the container with id a1d2f7ab1ee7f919615891e0dbee5b6722050acfbab408e8ca06450d54ca608b Nov 03 10:49:38.836679 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:38.836654 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw"] Nov 03 10:49:39.166355 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:39.166324 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls"] Nov 03 10:49:39.166451 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:39.166415 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebe5575_ee5c_448e_8858_0c6e3803efe1.slice/crio-b89dd5259f83bb1db7dbfbaa81c1a8d324c019238ec8ab42995615a9258bf9de WatchSource:0}: Error finding container b89dd5259f83bb1db7dbfbaa81c1a8d324c019238ec8ab42995615a9258bf9de: Status 404 returned error can't find the container with id b89dd5259f83bb1db7dbfbaa81c1a8d324c019238ec8ab42995615a9258bf9de Nov 03 10:49:39.244834 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:39.244797 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" event={"ID":"3ebe5575-ee5c-448e-8858-0c6e3803efe1","Type":"ContainerStarted","Data":"b89dd5259f83bb1db7dbfbaa81c1a8d324c019238ec8ab42995615a9258bf9de"} Nov 03 10:49:39.247228 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:39.247203 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" event={"ID":"046018fb-6293-435f-a62e-9d1fccde20b3","Type":"ContainerStarted","Data":"6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83"} Nov 03 10:49:39.247387 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:39.247253 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" event={"ID":"046018fb-6293-435f-a62e-9d1fccde20b3","Type":"ContainerStarted","Data":"a1d2f7ab1ee7f919615891e0dbee5b6722050acfbab408e8ca06450d54ca608b"} Nov 03 10:49:39.247387 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:39.247283 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:39.267479 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:39.267423 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" podStartSLOduration=1.267410027 podStartE2EDuration="1.267410027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:39.263987801 +0000 UTC m=+1257.248859083" watchObservedRunningTime="2026-11-03 10:49:39.267410027 +0000 UTC m=+1257.252281317" Nov 03 10:49:40.257354 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:40.257273 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" event={"ID":"3ebe5575-ee5c-448e-8858-0c6e3803efe1","Type":"ContainerStarted","Data":"45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076"} Nov 03 10:49:40.280375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:40.280325 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" podStartSLOduration=2.280309892 podStartE2EDuration="2.280309892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:40.277039909 +0000 UTC m=+1258.261911193" watchObservedRunningTime="2026-11-03 10:49:40.280309892 +0000 UTC m=+1258.265181166" Nov 03 10:49:41.321318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:41.313967 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:43.069170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:43.069139 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:49:48.532246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:48.532049 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl"] Nov 03 10:49:48.557173 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:48.557126 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn"] Nov 03 10:49:48.626150 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:48.626105 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl"] Nov 03 10:49:48.626470 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:48.626439 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" containerID="cri-o://96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199" gracePeriod=30 Nov 03 10:49:48.632404 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:48.632379 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7"] Nov 03 10:49:48.632973 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:48.632908 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" containerID="cri-o://33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994" gracePeriod=30 Nov 03 10:49:48.640457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:48.640426 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8"] Nov 03 10:49:48.640716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:48.640689 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" containerID="cri-o://e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b" gracePeriod=30 Nov 03 10:49:50.273991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:50.271960 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:49:51.326696 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.326653 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5"] Nov 03 10:49:51.334093 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.334066 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:49:51.339357 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.339325 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5"] Nov 03 10:49:51.339900 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.339877 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4d9549-dockercfg-przpl\"" Nov 03 10:49:51.382734 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.382701 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69tmc\" (UniqueName: \"kubernetes.io/projected/f21e89d4-3943-4edb-ad24-4718c7f19ed1-kube-api-access-69tmc\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5\" (UID: \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:49:51.382929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.382847 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21e89d4-3943-4edb-ad24-4718c7f19ed1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5\" (UID: \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:49:51.486418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.484453 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69tmc\" (UniqueName: \"kubernetes.io/projected/f21e89d4-3943-4edb-ad24-4718c7f19ed1-kube-api-access-69tmc\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5\" (UID: \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:49:51.486418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.484766 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21e89d4-3943-4edb-ad24-4718c7f19ed1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5\" (UID: \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:49:51.494869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.494827 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21e89d4-3943-4edb-ad24-4718c7f19ed1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5\" (UID: \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:49:51.553289 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.552621 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69tmc\" (UniqueName: \"kubernetes.io/projected/f21e89d4-3943-4edb-ad24-4718c7f19ed1-kube-api-access-69tmc\") pod \"payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5\" (UID: \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:49:51.651482 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.651390 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:49:51.936760 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:51.936726 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5"] Nov 03 10:49:51.936946 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:49:51.936923 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21e89d4_3943_4edb_ad24_4718c7f19ed1.slice/crio-3c9a4c6465c1be03f91df959e2e21ed8bd5f9519a67178d43ff44981249a368e WatchSource:0}: Error finding container 3c9a4c6465c1be03f91df959e2e21ed8bd5f9519a67178d43ff44981249a368e: Status 404 returned error can't find the container with id 3c9a4c6465c1be03f91df959e2e21ed8bd5f9519a67178d43ff44981249a368e Nov 03 10:49:52.316766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:52.316734 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:49:52.434280 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:52.434245 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" event={"ID":"f21e89d4-3943-4edb-ad24-4718c7f19ed1","Type":"ContainerStarted","Data":"62244394fec9d83711f0f1e857299bdb3fc80736e465f305f8f228ae2986c712"} Nov 03 10:49:52.434983 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:52.434357 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:49:52.434983 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:52.434377 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" event={"ID":"f21e89d4-3943-4edb-ad24-4718c7f19ed1","Type":"ContainerStarted","Data":"3c9a4c6465c1be03f91df959e2e21ed8bd5f9519a67178d43ff44981249a368e"} Nov 03 10:49:52.451388 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:52.451324 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" podStartSLOduration=1.451308644 podStartE2EDuration="1.451308644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:52.449364139 +0000 UTC m=+1270.434235425" watchObservedRunningTime="2026-11-03 10:49:52.451308644 +0000 UTC m=+1270.436179918" Nov 03 10:49:52.546080 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:52.546036 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 10:49:52.546245 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:52.546117 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 10:49:52.638326 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:52.638236 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 10:49:52.638326 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:52.638309 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 10:49:53.532664 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.532615 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" podUID="b43b6405-ac5b-46af-992c-c021aeb9a908" containerName="payload-processing" containerID="cri-o://aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d" gracePeriod=25 Nov 03 10:49:53.558038 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.557470 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" podUID="9b75cf94-7724-4b8f-b42c-42aa91b25bd4" containerName="payload-pre-processing" containerID="cri-o://201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a" gracePeriod=25 Nov 03 10:49:53.670658 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.670626 5086 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 10:49:53.670793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.670698 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 10:49:53.866909 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.866658 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:53.928932 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.927460 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b43b6405-ac5b-46af-992c-c021aeb9a908-kube-api-access-4pw5n\" (UniqueName: \"kubernetes.io/projected/b43b6405-ac5b-46af-992c-c021aeb9a908-kube-api-access-4pw5n\") pod \"b43b6405-ac5b-46af-992c-c021aeb9a908\" (UID: \"b43b6405-ac5b-46af-992c-c021aeb9a908\") " Nov 03 10:49:53.928932 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.927946 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b43b6405-ac5b-46af-992c-c021aeb9a908-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43b6405-ac5b-46af-992c-c021aeb9a908-plugins-config-volume\") pod \"b43b6405-ac5b-46af-992c-c021aeb9a908\" (UID: \"b43b6405-ac5b-46af-992c-c021aeb9a908\") " Nov 03 10:49:53.928932 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.928748 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43b6405-ac5b-46af-992c-c021aeb9a908-plugins-config-volume" pod "b43b6405-ac5b-46af-992c-c021aeb9a908" (UID: "b43b6405-ac5b-46af-992c-c021aeb9a908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:53.944549 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.944513 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43b6405-ac5b-46af-992c-c021aeb9a908-kube-api-access-4pw5n" pod "b43b6405-ac5b-46af-992c-c021aeb9a908" (UID: "b43b6405-ac5b-46af-992c-c021aeb9a908"). InnerVolumeSpecName "kube-api-access-4pw5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:53.977580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:53.977550 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:54.038513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.038481 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43b6405-ac5b-46af-992c-c021aeb9a908-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.038513 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.038517 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pw5n\" (UniqueName: \"kubernetes.io/projected/b43b6405-ac5b-46af-992c-c021aeb9a908-kube-api-access-4pw5n\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.059026 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.058995 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:49:54.062168 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.062146 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:49:54.070115 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.070092 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:49:54.145944 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.145900 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-socket\") pod \"6db155cc-6494-41bc-8c24-b8bc6addfae6\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " Nov 03 10:49:54.147018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.146987 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-kube-api-access-rkblf\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-kube-api-access-rkblf\") pod \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " Nov 03 10:49:54.147159 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147039 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-plugins-config-volume\") pod \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\" (UID: \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\") " Nov 03 10:49:54.147159 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147096 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-certs\") pod \"6db155cc-6494-41bc-8c24-b8bc6addfae6\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " Nov 03 10:49:54.147276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147155 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-kube-api-access-czgzn\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-kube-api-access-czgzn\") pod \"aafe9657-ae56-48e9-b18d-096c0180cf28\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " Nov 03 10:49:54.147276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147258 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-socket\") pod \"aafe9657-ae56-48e9-b18d-096c0180cf28\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " Nov 03 10:49:54.147524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147505 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-envoy\") pod \"aafe9657-ae56-48e9-b18d-096c0180cf28\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " Nov 03 10:49:54.147609 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147549 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-certs\") pod \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " Nov 03 10:49:54.147669 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147607 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-kube-api-access-28wqw\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-kube-api-access-28wqw\") pod \"6db155cc-6494-41bc-8c24-b8bc6addfae6\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " Nov 03 10:49:54.147669 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147649 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-certs\") pod \"aafe9657-ae56-48e9-b18d-096c0180cf28\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " Nov 03 10:49:54.147776 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147684 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-credential-socket\") pod \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " Nov 03 10:49:54.147839 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147781 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-podinfo\") pod \"aafe9657-ae56-48e9-b18d-096c0180cf28\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " Nov 03 10:49:54.147839 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147820 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istiod-ca-cert\") pod \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " Nov 03 10:49:54.147944 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147848 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-envoy\") pod \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " Nov 03 10:49:54.147944 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147897 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-envoy\") pod \"6db155cc-6494-41bc-8c24-b8bc6addfae6\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " Nov 03 10:49:54.147944 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147930 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-kube-api-access-8hpvj\" (UniqueName: \"kubernetes.io/projected/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-kube-api-access-8hpvj\") pod \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\" (UID: \"9b75cf94-7724-4b8f-b42c-42aa91b25bd4\") " Nov 03 10:49:54.148121 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147960 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-data\") pod \"6db155cc-6494-41bc-8c24-b8bc6addfae6\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " Nov 03 10:49:54.148121 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.147989 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db155cc-6494-41bc-8c24-b8bc6addfae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db155cc-6494-41bc-8c24-b8bc6addfae6-istiod-ca-cert\") pod \"6db155cc-6494-41bc-8c24-b8bc6addfae6\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " Nov 03 10:49:54.148121 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148027 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafe9657-ae56-48e9-b18d-096c0180cf28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe9657-ae56-48e9-b18d-096c0180cf28-istiod-ca-cert\") pod \"aafe9657-ae56-48e9-b18d-096c0180cf28\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " Nov 03 10:49:54.148121 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148062 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-podinfo\") pod \"6db155cc-6494-41bc-8c24-b8bc6addfae6\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " Nov 03 10:49:54.148379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148119 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-socket" pod "6db155cc-6494-41bc-8c24-b8bc6addfae6" (UID: "6db155cc-6494-41bc-8c24-b8bc6addfae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.148379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148131 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-token\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-token\") pod \"aafe9657-ae56-48e9-b18d-096c0180cf28\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " Nov 03 10:49:54.148379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148196 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-podinfo\") pod \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " Nov 03 10:49:54.148379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148238 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-data\") pod \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " Nov 03 10:49:54.148379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148278 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-credential-socket\") pod \"aafe9657-ae56-48e9-b18d-096c0180cf28\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " Nov 03 10:49:54.148379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148346 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-token\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-token\") pod \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " Nov 03 10:49:54.148744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148427 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-credential-socket\") pod \"6db155cc-6494-41bc-8c24-b8bc6addfae6\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " Nov 03 10:49:54.148744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148457 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-token\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-token\") pod \"6db155cc-6494-41bc-8c24-b8bc6addfae6\" (UID: \"6db155cc-6494-41bc-8c24-b8bc6addfae6\") " Nov 03 10:49:54.148744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148528 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-socket\") pod \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\" (UID: \"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6\") " Nov 03 10:49:54.148744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.148561 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-data\") pod \"aafe9657-ae56-48e9-b18d-096c0180cf28\" (UID: \"aafe9657-ae56-48e9-b18d-096c0180cf28\") " Nov 03 10:49:54.149361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.149335 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-data" pod "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" (UID: "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.149476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.149373 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-credential-socket" pod "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" (UID: "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.149685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.149659 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-credential-socket" pod "6db155cc-6494-41bc-8c24-b8bc6addfae6" (UID: "6db155cc-6494-41bc-8c24-b8bc6addfae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.149969 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.149949 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-socket" pod "aafe9657-ae56-48e9-b18d-096c0180cf28" (UID: "aafe9657-ae56-48e9-b18d-096c0180cf28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.150110 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.149994 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-credential-socket" pod "aafe9657-ae56-48e9-b18d-096c0180cf28" (UID: "aafe9657-ae56-48e9-b18d-096c0180cf28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.150236 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.150212 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-data" pod "aafe9657-ae56-48e9-b18d-096c0180cf28" (UID: "aafe9657-ae56-48e9-b18d-096c0180cf28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.150335 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.150226 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-plugins-config-volume" pod "9b75cf94-7724-4b8f-b42c-42aa91b25bd4" (UID: "9b75cf94-7724-4b8f-b42c-42aa91b25bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:54.150501 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.150478 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-socket" pod "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" (UID: "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.150569 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.150509 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-certs" pod "6db155cc-6494-41bc-8c24-b8bc6addfae6" (UID: "6db155cc-6494-41bc-8c24-b8bc6addfae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.150569 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.150555 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.150939 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.150914 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db155cc-6494-41bc-8c24-b8bc6addfae6-istiod-ca-cert" pod "6db155cc-6494-41bc-8c24-b8bc6addfae6" (UID: "6db155cc-6494-41bc-8c24-b8bc6addfae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:54.153423 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.153394 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-data" pod "6db155cc-6494-41bc-8c24-b8bc6addfae6" (UID: "6db155cc-6494-41bc-8c24-b8bc6addfae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.153727 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.153641 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istiod-ca-cert" pod "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" (UID: "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:54.155383 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.155111 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafe9657-ae56-48e9-b18d-096c0180cf28-istiod-ca-cert" pod "aafe9657-ae56-48e9-b18d-096c0180cf28" (UID: "aafe9657-ae56-48e9-b18d-096c0180cf28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:49:54.156575 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.156407 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-kube-api-access-8hpvj" pod "9b75cf94-7724-4b8f-b42c-42aa91b25bd4" (UID: "9b75cf94-7724-4b8f-b42c-42aa91b25bd4"). InnerVolumeSpecName "kube-api-access-8hpvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:54.157015 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.156985 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-podinfo" pod "aafe9657-ae56-48e9-b18d-096c0180cf28" (UID: "aafe9657-ae56-48e9-b18d-096c0180cf28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:49:54.158312 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.158273 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-certs" pod "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" (UID: "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.159180 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.159154 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-podinfo" pod "6db155cc-6494-41bc-8c24-b8bc6addfae6" (UID: "6db155cc-6494-41bc-8c24-b8bc6addfae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:49:54.160148 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.159877 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-envoy" pod "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" (UID: "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.160971 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.160945 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-envoy" pod "aafe9657-ae56-48e9-b18d-096c0180cf28" (UID: "aafe9657-ae56-48e9-b18d-096c0180cf28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.161233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.161213 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-certs" pod "aafe9657-ae56-48e9-b18d-096c0180cf28" (UID: "aafe9657-ae56-48e9-b18d-096c0180cf28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.161512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.161494 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-envoy" pod "6db155cc-6494-41bc-8c24-b8bc6addfae6" (UID: "6db155cc-6494-41bc-8c24-b8bc6addfae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:49:54.162266 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.162227 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-kube-api-access-28wqw" pod "6db155cc-6494-41bc-8c24-b8bc6addfae6" (UID: "6db155cc-6494-41bc-8c24-b8bc6addfae6"). InnerVolumeSpecName "kube-api-access-28wqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:54.164631 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.164266 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-token" pod "6db155cc-6494-41bc-8c24-b8bc6addfae6" (UID: "6db155cc-6494-41bc-8c24-b8bc6addfae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:54.164631 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.164428 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-token" pod "aafe9657-ae56-48e9-b18d-096c0180cf28" (UID: "aafe9657-ae56-48e9-b18d-096c0180cf28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:54.165146 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.165127 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-kube-api-access-rkblf" pod "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" (UID: "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6"). InnerVolumeSpecName "kube-api-access-rkblf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:54.167328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.166859 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-podinfo" pod "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" (UID: "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:49:54.167328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.167209 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-token" pod "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" (UID: "56664a6c-c1b4-4e9d-a3de-0b50d2456fa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:54.168018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.167994 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-kube-api-access-czgzn" pod "aafe9657-ae56-48e9-b18d-096c0180cf28" (UID: "aafe9657-ae56-48e9-b18d-096c0180cf28"). InnerVolumeSpecName "kube-api-access-czgzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:49:54.263073 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.263039 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.263597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.263578 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.263721 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.263707 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28wqw\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-kube-api-access-28wqw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.263825 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.263812 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.263914 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.263903 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.263998 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.263987 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264090 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264078 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264170 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264159 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264239 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264350 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264338 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hpvj\" (UniqueName: \"kubernetes.io/projected/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-kube-api-access-8hpvj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264439 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264428 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264506 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db155cc-6494-41bc-8c24-b8bc6addfae6-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264639 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264626 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe9657-ae56-48e9-b18d-096c0180cf28-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264715 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264705 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264781 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264870 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264859 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.264951 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.264939 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265028 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265018 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265095 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265175 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265267 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265255 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db155cc-6494-41bc-8c24-b8bc6addfae6-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265356 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265500 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265490 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265569 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkblf\" (UniqueName: \"kubernetes.io/projected/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6-kube-api-access-rkblf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265643 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b75cf94-7724-4b8f-b42c-42aa91b25bd4-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265727 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265717 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db155cc-6494-41bc-8c24-b8bc6addfae6-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265799 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265788 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czgzn\" (UniqueName: \"kubernetes.io/projected/aafe9657-ae56-48e9-b18d-096c0180cf28-kube-api-access-czgzn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.265877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.265866 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe9657-ae56-48e9-b18d-096c0180cf28-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:49:54.299359 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:54.299229 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:49:54.454980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.454938 5086 generic.go:356] "Generic (PLEG): container finished" podID="b43b6405-ac5b-46af-992c-c021aeb9a908" containerID="aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d" exitCode=0 Nov 03 10:49:54.455180 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.454991 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" Nov 03 10:49:54.455370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.454989 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" event={"ID":"b43b6405-ac5b-46af-992c-c021aeb9a908","Type":"ContainerDied","Data":"aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d"} Nov 03 10:49:54.455486 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.455419 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl" event={"ID":"b43b6405-ac5b-46af-992c-c021aeb9a908","Type":"ContainerDied","Data":"f6b3d975de8f12f66eaebb3e2ce41fa82f2c6dfacb74bdb151963d58f50c0d8a"} Nov 03 10:49:54.455486 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.455454 5086 scope.go:122] "RemoveContainer" containerID="aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d" Nov 03 10:49:54.463212 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.462694 5086 generic.go:356] "Generic (PLEG): container finished" podID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerID="e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b" exitCode=0 Nov 03 10:49:54.463914 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.463886 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" event={"ID":"aafe9657-ae56-48e9-b18d-096c0180cf28","Type":"ContainerDied","Data":"e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b"} Nov 03 10:49:54.486764 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.473190 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" Nov 03 10:49:54.488393 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.488349 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8" event={"ID":"aafe9657-ae56-48e9-b18d-096c0180cf28","Type":"ContainerDied","Data":"a19eb83a05f839e0db401d9304d19469b6c285f076a335ae3eaf75758613a8ba"} Nov 03 10:49:54.488582 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.481399 5086 scope.go:122] "RemoveContainer" containerID="aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d" Nov 03 10:49:54.494787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.494752 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl"] Nov 03 10:49:54.501826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.501785 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-797857cb67-7jprl"] Nov 03 10:49:54.503157 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:54.503123 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d\": container with ID starting with aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d not found: ID does not exist" containerID="aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d" Nov 03 10:49:54.503949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.503903 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d"} err="failed to get container status \"aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d\": rpc error: code = NotFound desc = could not find container \"aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d\": container with ID starting with aaffb7d59c64474ebe3e84cf4fc9d885297c37c6dcb4df6261737447b83f2e5d not found: ID does not exist" Nov 03 10:49:54.504548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.504527 5086 scope.go:122] "RemoveContainer" containerID="e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b" Nov 03 10:49:54.505550 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.505525 5086 generic.go:356] "Generic (PLEG): container finished" podID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerID="33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994" exitCode=0 Nov 03 10:49:54.505963 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.505939 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" event={"ID":"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6","Type":"ContainerDied","Data":"33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994"} Nov 03 10:49:54.506141 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.506115 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" event={"ID":"56664a6c-c1b4-4e9d-a3de-0b50d2456fa6","Type":"ContainerDied","Data":"30d4b0045a1680b60518731ec672579ef131ed6d9f36e74b137da33b21680776"} Nov 03 10:49:54.506370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.506353 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7" Nov 03 10:49:54.510497 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.510472 5086 generic.go:356] "Generic (PLEG): container finished" podID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerID="96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199" exitCode=0 Nov 03 10:49:54.510740 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.510720 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" event={"ID":"6db155cc-6494-41bc-8c24-b8bc6addfae6","Type":"ContainerDied","Data":"96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199"} Nov 03 10:49:54.510899 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.510881 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" event={"ID":"6db155cc-6494-41bc-8c24-b8bc6addfae6","Type":"ContainerDied","Data":"d58d16a3eabb3be20b3eff46e504dbbf2b97617757d9b2b574b80467e18c7944"} Nov 03 10:49:54.511111 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.511095 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl" Nov 03 10:49:54.581207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.581071 5086 generic.go:356] "Generic (PLEG): container finished" podID="9b75cf94-7724-4b8f-b42c-42aa91b25bd4" containerID="201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a" exitCode=0 Nov 03 10:49:54.581746 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.581718 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" event={"ID":"9b75cf94-7724-4b8f-b42c-42aa91b25bd4","Type":"ContainerDied","Data":"201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a"} Nov 03 10:49:54.581911 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.581892 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" event={"ID":"9b75cf94-7724-4b8f-b42c-42aa91b25bd4","Type":"ContainerDied","Data":"7f880f69beb01d752d2f142236432e7e2b333519ebebe14ef8a33638bd9de169"} Nov 03 10:49:54.582077 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.582063 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn" Nov 03 10:49:54.644194 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.644146 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd"] Nov 03 10:49:54.661842 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.661814 5086 scope.go:122] "RemoveContainer" containerID="e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b" Nov 03 10:49:54.664625 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:54.664028 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b\": container with ID starting with e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b not found: ID does not exist" containerID="e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b" Nov 03 10:49:54.664837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.664805 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b"} err="failed to get container status \"e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b\": rpc error: code = NotFound desc = could not find container \"e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b\": container with ID starting with e70bfa29cd7e45bb54ba7134d1f3f7a33d8be339307adbe6841fcb345ce4a89b not found: ID does not exist" Nov 03 10:49:54.664971 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.664958 5086 scope.go:122] "RemoveContainer" containerID="33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994" Nov 03 10:49:54.677913 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.677885 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7"] Nov 03 10:49:54.683968 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.683938 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-hprq7"] Nov 03 10:49:54.694612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.693567 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn"] Nov 03 10:49:54.708284 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.708243 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-5bf4978c5-4sdbn"] Nov 03 10:49:54.717786 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.717739 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8"] Nov 03 10:49:54.756394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.756358 5086 scope.go:122] "RemoveContainer" containerID="33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994" Nov 03 10:49:54.757068 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:54.757032 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994\": container with ID starting with 33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994 not found: ID does not exist" containerID="33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994" Nov 03 10:49:54.757175 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.757083 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994"} err="failed to get container status \"33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994\": rpc error: code = NotFound desc = could not find container \"33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994\": container with ID starting with 33938d0191629e5f04adbdeb608768e9da3eb30aab4938d691a1d920d4f7c994 not found: ID does not exist" Nov 03 10:49:54.757175 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.757111 5086 scope.go:122] "RemoveContainer" containerID="96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199" Nov 03 10:49:54.761284 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.761264 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-4j5n8"] Nov 03 10:49:54.769063 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.769001 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl"] Nov 03 10:49:54.775753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.775730 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wvlbl"] Nov 03 10:49:54.783583 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.783561 5086 scope.go:122] "RemoveContainer" containerID="96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199" Nov 03 10:49:54.784088 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:54.783962 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199\": container with ID starting with 96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199 not found: ID does not exist" containerID="96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199" Nov 03 10:49:54.784088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.783994 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199"} err="failed to get container status \"96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199\": rpc error: code = NotFound desc = could not find container \"96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199\": container with ID starting with 96ff50364bf36672b9d3896aaa34d7189216d1fb1a9ba31dd47775f3b47ae199 not found: ID does not exist" Nov 03 10:49:54.784088 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.784013 5086 scope.go:122] "RemoveContainer" containerID="201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a" Nov 03 10:49:54.795535 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.795515 5086 scope.go:122] "RemoveContainer" containerID="201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a" Nov 03 10:49:54.795856 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:49:54.795835 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a\": container with ID starting with 201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a not found: ID does not exist" containerID="201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a" Nov 03 10:49:54.795922 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:54.795866 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a"} err="failed to get container status \"201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a\": rpc error: code = NotFound desc = could not find container \"201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a\": container with ID starting with 201875bff451781e4d19e1acd743c40b86c3030595ea462df122b3253fdf580a not found: ID does not exist" Nov 03 10:49:56.178771 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:56.178712 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" path="/var/lib/kubelet/pods/56664a6c-c1b4-4e9d-a3de-0b50d2456fa6/volumes" Nov 03 10:49:56.179421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:56.179400 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" path="/var/lib/kubelet/pods/6db155cc-6494-41bc-8c24-b8bc6addfae6/volumes" Nov 03 10:49:56.179875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:56.179855 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b75cf94-7724-4b8f-b42c-42aa91b25bd4" path="/var/lib/kubelet/pods/9b75cf94-7724-4b8f-b42c-42aa91b25bd4/volumes" Nov 03 10:49:56.180705 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:56.180685 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" path="/var/lib/kubelet/pods/aafe9657-ae56-48e9-b18d-096c0180cf28/volumes" Nov 03 10:49:56.181121 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:56.181101 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b43b6405-ac5b-46af-992c-c021aeb9a908" path="/var/lib/kubelet/pods/b43b6405-ac5b-46af-992c-c021aeb9a908/volumes" Nov 03 10:49:59.659521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:49:59.659449 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" podUID="1b05c6e5-6d6b-4f96-84d0-3d85a35efb67" containerName="payload-pre-processing" containerID="cri-o://c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535" gracePeriod=25 Nov 03 10:50:00.054523 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.054486 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:50:00.188561 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.188529 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-plugins-config-volume\") pod \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\" (UID: \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\") " Nov 03 10:50:00.188767 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.188649 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-kube-api-access-dksw7\" (UniqueName: \"kubernetes.io/projected/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-kube-api-access-dksw7\") pod \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\" (UID: \"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67\") " Nov 03 10:50:00.189033 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.189001 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-plugins-config-volume" pod "1b05c6e5-6d6b-4f96-84d0-3d85a35efb67" (UID: "1b05c6e5-6d6b-4f96-84d0-3d85a35efb67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:50:00.189377 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.189350 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:00.190961 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.190937 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-kube-api-access-dksw7" pod "1b05c6e5-6d6b-4f96-84d0-3d85a35efb67" (UID: "1b05c6e5-6d6b-4f96-84d0-3d85a35efb67"). InnerVolumeSpecName "kube-api-access-dksw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:50:00.296378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.292778 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dksw7\" (UniqueName: \"kubernetes.io/projected/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67-kube-api-access-dksw7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:00.654230 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.654194 5086 generic.go:356] "Generic (PLEG): container finished" podID="1b05c6e5-6d6b-4f96-84d0-3d85a35efb67" containerID="c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535" exitCode=0 Nov 03 10:50:00.654434 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.654258 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" Nov 03 10:50:00.654434 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.654250 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" event={"ID":"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67","Type":"ContainerDied","Data":"c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535"} Nov 03 10:50:00.654434 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.654403 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd" event={"ID":"1b05c6e5-6d6b-4f96-84d0-3d85a35efb67","Type":"ContainerDied","Data":"0f0e9fe20ded38f5eba8343c2b5f443258f6240e697cf7f987c781fcb9d61dfb"} Nov 03 10:50:00.654615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.654436 5086 scope.go:122] "RemoveContainer" containerID="c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535" Nov 03 10:50:00.667917 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.667899 5086 scope.go:122] "RemoveContainer" containerID="c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535" Nov 03 10:50:00.668357 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:50:00.668233 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535\": container with ID starting with c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535 not found: ID does not exist" containerID="c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535" Nov 03 10:50:00.668357 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.668259 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535"} err="failed to get container status \"c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535\": rpc error: code = NotFound desc = could not find container \"c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535\": container with ID starting with c324e646e87223d0c58d0f8beee7698bae7535da0c6895b95f15b8a063b0f535 not found: ID does not exist" Nov 03 10:50:00.679708 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.679679 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd"] Nov 03 10:50:00.688275 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:00.688216 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-6cf854f859-rbvgd"] Nov 03 10:50:02.178520 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.178479 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b05c6e5-6d6b-4f96-84d0-3d85a35efb67" path="/var/lib/kubelet/pods/1b05c6e5-6d6b-4f96-84d0-3d85a35efb67/volumes" Nov 03 10:50:02.292931 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.292899 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr"] Nov 03 10:50:02.302652 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.302618 5086 cpu_manager.go:414] "containerMap: removing container" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" Nov 03 10:50:02.302873 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.302854 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" Nov 03 10:50:02.302986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.302975 5086 cpu_manager.go:414] "containerMap: removing container" podUID="9b75cf94-7724-4b8f-b42c-42aa91b25bd4" containerName="payload-pre-processing" Nov 03 10:50:02.303062 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.303051 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b75cf94-7724-4b8f-b42c-42aa91b25bd4" containerName="payload-pre-processing" Nov 03 10:50:02.303163 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.303152 5086 cpu_manager.go:414] "containerMap: removing container" podUID="b43b6405-ac5b-46af-992c-c021aeb9a908" containerName="payload-processing" Nov 03 10:50:02.303245 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.303235 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43b6405-ac5b-46af-992c-c021aeb9a908" containerName="payload-processing" Nov 03 10:50:02.303345 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.303334 5086 cpu_manager.go:414] "containerMap: removing container" podUID="1b05c6e5-6d6b-4f96-84d0-3d85a35efb67" containerName="payload-pre-processing" Nov 03 10:50:02.303429 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.303417 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b05c6e5-6d6b-4f96-84d0-3d85a35efb67" containerName="payload-pre-processing" Nov 03 10:50:02.303525 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.303515 5086 cpu_manager.go:414] "containerMap: removing container" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" Nov 03 10:50:02.303595 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.303585 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" Nov 03 10:50:02.303714 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.303703 5086 cpu_manager.go:414] "containerMap: removing container" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" Nov 03 10:50:02.303789 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.303778 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" Nov 03 10:50:02.304113 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.304095 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafe9657-ae56-48e9-b18d-096c0180cf28" containerName="istio-proxy" Nov 03 10:50:02.309530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.309470 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b75cf94-7724-4b8f-b42c-42aa91b25bd4" containerName="payload-pre-processing" Nov 03 10:50:02.309886 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.309869 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43b6405-ac5b-46af-992c-c021aeb9a908" containerName="payload-processing" Nov 03 10:50:02.309985 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.309975 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b05c6e5-6d6b-4f96-84d0-3d85a35efb67" containerName="payload-pre-processing" Nov 03 10:50:02.310063 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.310054 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="56664a6c-c1b4-4e9d-a3de-0b50d2456fa6" containerName="istio-proxy" Nov 03 10:50:02.310150 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.310142 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db155cc-6494-41bc-8c24-b8bc6addfae6" containerName="istio-proxy" Nov 03 10:50:02.324054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.320133 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:02.361144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.361103 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfhj\" (UniqueName: \"kubernetes.io/projected/4b44b98a-26df-4b24-a982-bb77a7bc12d4-kube-api-access-mmfhj\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr\" (UID: \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:02.386320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.385099 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b44b98a-26df-4b24-a982-bb77a7bc12d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr\" (UID: \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:02.386320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.384084 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aa420c-dockercfg-lj7vq\"" Nov 03 10:50:02.386320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.384185 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa420c\"" Nov 03 10:50:02.425326 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.421876 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr"] Nov 03 10:50:02.497993 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.496175 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b44b98a-26df-4b24-a982-bb77a7bc12d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr\" (UID: \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:02.497993 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.496557 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfhj\" (UniqueName: \"kubernetes.io/projected/4b44b98a-26df-4b24-a982-bb77a7bc12d4-kube-api-access-mmfhj\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr\" (UID: \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:02.497993 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.497949 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b44b98a-26df-4b24-a982-bb77a7bc12d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr\" (UID: \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:02.508977 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.508935 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh"] Nov 03 10:50:02.512447 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.512418 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfhj\" (UniqueName: \"kubernetes.io/projected/4b44b98a-26df-4b24-a982-bb77a7bc12d4-kube-api-access-mmfhj\") pod \"payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr\" (UID: \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:02.515044 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.515018 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh"] Nov 03 10:50:02.515436 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.515418 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:02.661456 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.615190 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679be1a2-3c56-467f-940c-fac577127482-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh\" (UID: \"679be1a2-3c56-467f-940c-fac577127482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:02.661456 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.615274 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf69m\" (UniqueName: \"kubernetes.io/projected/679be1a2-3c56-467f-940c-fac577127482-kube-api-access-tf69m\") pod \"payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh\" (UID: \"679be1a2-3c56-467f-940c-fac577127482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:02.734856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.732481 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679be1a2-3c56-467f-940c-fac577127482-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh\" (UID: \"679be1a2-3c56-467f-940c-fac577127482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:02.734856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.732574 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf69m\" (UniqueName: \"kubernetes.io/projected/679be1a2-3c56-467f-940c-fac577127482-kube-api-access-tf69m\") pod \"payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh\" (UID: \"679be1a2-3c56-467f-940c-fac577127482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:02.734856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.733399 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:02.762835 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.762763 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679be1a2-3c56-467f-940c-fac577127482-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh\" (UID: \"679be1a2-3c56-467f-940c-fac577127482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:02.786267 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.786229 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf69m\" (UniqueName: \"kubernetes.io/projected/679be1a2-3c56-467f-940c-fac577127482-kube-api-access-tf69m\") pod \"payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh\" (UID: \"679be1a2-3c56-467f-940c-fac577127482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:02.894319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:02.890557 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:03.483322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:03.473456 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:50:03.698944 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:50:03.698892 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b44b98a_26df_4b24_a982_bb77a7bc12d4.slice/crio-447b41b1c45b156d96618ac62a2f8fd5cfeede674441b8ef197d868b0ad5773e WatchSource:0}: Error finding container 447b41b1c45b156d96618ac62a2f8fd5cfeede674441b8ef197d868b0ad5773e: Status 404 returned error can't find the container with id 447b41b1c45b156d96618ac62a2f8fd5cfeede674441b8ef197d868b0ad5773e Nov 03 10:50:03.702663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:03.702632 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr"] Nov 03 10:50:03.814255 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:03.814217 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" event={"ID":"4b44b98a-26df-4b24-a982-bb77a7bc12d4","Type":"ContainerStarted","Data":"447b41b1c45b156d96618ac62a2f8fd5cfeede674441b8ef197d868b0ad5773e"} Nov 03 10:50:04.009784 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.009730 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh"] Nov 03 10:50:04.104956 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:50:04.104928 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679be1a2_3c56_467f_940c_fac577127482.slice/crio-58a9654963eb04fddd2fee2f41a775e8e82a1270be1bdee8fe4036a66826497e WatchSource:0}: Error finding container 58a9654963eb04fddd2fee2f41a775e8e82a1270be1bdee8fe4036a66826497e: Status 404 returned error can't find the container with id 58a9654963eb04fddd2fee2f41a775e8e82a1270be1bdee8fe4036a66826497e Nov 03 10:50:04.234600 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.234560 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5p4bj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:50:04.234853 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.234820 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podUID="b7804ed8-ba49-4936-99f0-f02253369475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:50:04.249374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.244370 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lrmsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:50:04.249374 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.244420 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podUID="99bbba2f-874c-4246-9ec3-e92e117a4159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:50:04.826306 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.826256 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" event={"ID":"4b44b98a-26df-4b24-a982-bb77a7bc12d4","Type":"ContainerStarted","Data":"a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce"} Nov 03 10:50:04.826857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.826379 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:04.829254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.829222 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" event={"ID":"679be1a2-3c56-467f-940c-fac577127482","Type":"ContainerStarted","Data":"88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc"} Nov 03 10:50:04.829452 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.829277 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" event={"ID":"679be1a2-3c56-467f-940c-fac577127482","Type":"ContainerStarted","Data":"58a9654963eb04fddd2fee2f41a775e8e82a1270be1bdee8fe4036a66826497e"} Nov 03 10:50:04.829452 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.829396 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:04.849595 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.849530 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" podStartSLOduration=2.849273415 podStartE2EDuration="2.849273415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:04.844153125 +0000 UTC m=+1282.829024409" watchObservedRunningTime="2026-11-03 10:50:04.849273415 +0000 UTC m=+1282.834144704" Nov 03 10:50:04.866758 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:04.866522 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" podStartSLOduration=2.86650295 podStartE2EDuration="2.86650295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:04.861203533 +0000 UTC m=+1282.846074818" watchObservedRunningTime="2026-11-03 10:50:04.86650295 +0000 UTC m=+1282.851374238" Nov 03 10:50:06.232636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:06.230344 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj"] Nov 03 10:50:06.234330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:06.234284 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh"] Nov 03 10:50:11.230893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.230850 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" podUID="3692a26a-a911-44ea-b9fa-3c91af5d7748" containerName="payload-processing" containerID="cri-o://21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd" gracePeriod=25 Nov 03 10:50:11.235811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.235704 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" podUID="9feef9dd-d6c6-4cd0-8928-259d28809ce1" containerName="payload-pre-processing" containerID="cri-o://57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66" gracePeriod=25 Nov 03 10:50:11.571851 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.571822 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:50:11.600894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.600869 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:50:11.701992 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.701956 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9feef9dd-d6c6-4cd0-8928-259d28809ce1-kube-api-access-49nxq\" (UniqueName: \"kubernetes.io/projected/9feef9dd-d6c6-4cd0-8928-259d28809ce1-kube-api-access-49nxq\") pod \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\" (UID: \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\") " Nov 03 10:50:11.702168 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.702017 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3692a26a-a911-44ea-b9fa-3c91af5d7748-kube-api-access-c98t9\" (UniqueName: \"kubernetes.io/projected/3692a26a-a911-44ea-b9fa-3c91af5d7748-kube-api-access-c98t9\") pod \"3692a26a-a911-44ea-b9fa-3c91af5d7748\" (UID: \"3692a26a-a911-44ea-b9fa-3c91af5d7748\") " Nov 03 10:50:11.702168 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.702132 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9feef9dd-d6c6-4cd0-8928-259d28809ce1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feef9dd-d6c6-4cd0-8928-259d28809ce1-plugins-config-volume\") pod \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\" (UID: \"9feef9dd-d6c6-4cd0-8928-259d28809ce1\") " Nov 03 10:50:11.702168 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.702162 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3692a26a-a911-44ea-b9fa-3c91af5d7748-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3692a26a-a911-44ea-b9fa-3c91af5d7748-plugins-config-volume\") pod \"3692a26a-a911-44ea-b9fa-3c91af5d7748\" (UID: \"3692a26a-a911-44ea-b9fa-3c91af5d7748\") " Nov 03 10:50:11.702598 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.702568 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9feef9dd-d6c6-4cd0-8928-259d28809ce1-plugins-config-volume" pod "9feef9dd-d6c6-4cd0-8928-259d28809ce1" (UID: "9feef9dd-d6c6-4cd0-8928-259d28809ce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:50:11.702733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.702655 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3692a26a-a911-44ea-b9fa-3c91af5d7748-plugins-config-volume" pod "3692a26a-a911-44ea-b9fa-3c91af5d7748" (UID: "3692a26a-a911-44ea-b9fa-3c91af5d7748"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:50:11.702789 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.702742 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feef9dd-d6c6-4cd0-8928-259d28809ce1-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:11.702789 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.702759 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3692a26a-a911-44ea-b9fa-3c91af5d7748-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:11.704218 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.704191 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feef9dd-d6c6-4cd0-8928-259d28809ce1-kube-api-access-49nxq" pod "9feef9dd-d6c6-4cd0-8928-259d28809ce1" (UID: "9feef9dd-d6c6-4cd0-8928-259d28809ce1"). InnerVolumeSpecName "kube-api-access-49nxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:50:11.704218 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.704198 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3692a26a-a911-44ea-b9fa-3c91af5d7748-kube-api-access-c98t9" pod "3692a26a-a911-44ea-b9fa-3c91af5d7748" (UID: "3692a26a-a911-44ea-b9fa-3c91af5d7748"). InnerVolumeSpecName "kube-api-access-c98t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:50:11.806378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.806136 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98t9\" (UniqueName: \"kubernetes.io/projected/3692a26a-a911-44ea-b9fa-3c91af5d7748-kube-api-access-c98t9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:11.806378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.806174 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49nxq\" (UniqueName: \"kubernetes.io/projected/9feef9dd-d6c6-4cd0-8928-259d28809ce1-kube-api-access-49nxq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:11.887092 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.887053 5086 generic.go:356] "Generic (PLEG): container finished" podID="3692a26a-a911-44ea-b9fa-3c91af5d7748" containerID="21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd" exitCode=0 Nov 03 10:50:11.887414 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.887380 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" event={"ID":"3692a26a-a911-44ea-b9fa-3c91af5d7748","Type":"ContainerDied","Data":"21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd"} Nov 03 10:50:11.887544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.887469 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" event={"ID":"3692a26a-a911-44ea-b9fa-3c91af5d7748","Type":"ContainerDied","Data":"c364f605eecc25f7670338efdd9ff34a60920efa51ef3a5855f534a789c18b66"} Nov 03 10:50:11.887544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.887498 5086 scope.go:122] "RemoveContainer" containerID="21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd" Nov 03 10:50:11.887669 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.887654 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj" Nov 03 10:50:11.888406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.888374 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr"] Nov 03 10:50:11.889766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.889734 5086 cpu_manager.go:414] "containerMap: removing container" podUID="3692a26a-a911-44ea-b9fa-3c91af5d7748" containerName="payload-processing" Nov 03 10:50:11.889766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.889759 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3692a26a-a911-44ea-b9fa-3c91af5d7748" containerName="payload-processing" Nov 03 10:50:11.889922 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.889816 5086 cpu_manager.go:414] "containerMap: removing container" podUID="9feef9dd-d6c6-4cd0-8928-259d28809ce1" containerName="payload-pre-processing" Nov 03 10:50:11.889922 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.889824 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9feef9dd-d6c6-4cd0-8928-259d28809ce1" containerName="payload-pre-processing" Nov 03 10:50:11.890040 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.890015 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="3692a26a-a911-44ea-b9fa-3c91af5d7748" containerName="payload-processing" Nov 03 10:50:11.890040 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.890034 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="9feef9dd-d6c6-4cd0-8928-259d28809ce1" containerName="payload-pre-processing" Nov 03 10:50:11.891246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.891207 5086 generic.go:356] "Generic (PLEG): container finished" podID="9feef9dd-d6c6-4cd0-8928-259d28809ce1" containerID="57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66" exitCode=0 Nov 03 10:50:11.891367 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.891350 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" Nov 03 10:50:11.895041 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.894889 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" event={"ID":"9feef9dd-d6c6-4cd0-8928-259d28809ce1","Type":"ContainerDied","Data":"57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66"} Nov 03 10:50:11.895041 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.894948 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh" event={"ID":"9feef9dd-d6c6-4cd0-8928-259d28809ce1","Type":"ContainerDied","Data":"192e2ac96d33c92007c041d066b5793b79d75252cac6d0d81ca0c6d2ff6b843a"} Nov 03 10:50:11.895234 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.895059 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:11.910883 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.910866 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a2697c-serving-cert\"" Nov 03 10:50:11.920236 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.920152 5086 scope.go:122] "RemoveContainer" containerID="21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd" Nov 03 10:50:11.920820 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:50:11.920777 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd\": container with ID starting with 21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd not found: ID does not exist" containerID="21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd" Nov 03 10:50:11.921011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.920818 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd"} err="failed to get container status \"21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd\": rpc error: code = NotFound desc = could not find container \"21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd\": container with ID starting with 21556db63e14b9fc98f61e718fb6604a3468b0c5e2e5aca9713bcbbeb88cc8dd not found: ID does not exist" Nov 03 10:50:11.921011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.920843 5086 scope.go:122] "RemoveContainer" containerID="57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66" Nov 03 10:50:11.922072 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.922048 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr"] Nov 03 10:50:11.936466 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.936445 5086 scope.go:122] "RemoveContainer" containerID="57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66" Nov 03 10:50:11.936831 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:50:11.936806 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66\": container with ID starting with 57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66 not found: ID does not exist" containerID="57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66" Nov 03 10:50:11.936935 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.936854 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66"} err="failed to get container status \"57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66\": rpc error: code = NotFound desc = could not find container \"57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66\": container with ID starting with 57b8f0fa4f0301867e7cf8a9206f450acc4d4ea55950faf378ea02c1d2a5cc66 not found: ID does not exist" Nov 03 10:50:11.953339 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.953275 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj"] Nov 03 10:50:11.959754 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.959729 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-38c9b0-7dffb748cf-j4wcj"] Nov 03 10:50:11.967019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.966996 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh"] Nov 03 10:50:11.972722 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:11.972699 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-38c9b0-7974bd5d5f-bclmh"] Nov 03 10:50:12.015126 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.015086 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq76b\" (UniqueName: \"kubernetes.io/projected/f952ba2e-5049-4a57-851c-7ee424a7b75b-kube-api-access-zq76b\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.015376 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.015286 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.015376 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.015338 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-metrics-certs\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.015682 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.015661 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f952ba2e-5049-4a57-851c-7ee424a7b75b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.117766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.117723 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.117959 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.117775 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-metrics-certs\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.117959 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.117843 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f952ba2e-5049-4a57-851c-7ee424a7b75b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.117959 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:50:12.117866 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-serving-cert: secret "maas-api-e2e-ipp-b-a2697c-serving-cert" not found Nov 03 10:50:12.117959 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:50:12.117950 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls podName:f952ba2e-5049-4a57-851c-7ee424a7b75b nodeName:}" failed. No retries permitted until 2026-11-03 10:50:12.617929511 +0000 UTC m=+1290.602800790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls") pod "maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" (UID: "f952ba2e-5049-4a57-851c-7ee424a7b75b") : secret "maas-api-e2e-ipp-b-a2697c-serving-cert" not found Nov 03 10:50:12.118150 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.117976 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq76b\" (UniqueName: \"kubernetes.io/projected/f952ba2e-5049-4a57-851c-7ee424a7b75b-kube-api-access-zq76b\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.118611 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.118586 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f952ba2e-5049-4a57-851c-7ee424a7b75b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.120566 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.120544 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-metrics-certs\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.127874 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.127850 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq76b\" (UniqueName: \"kubernetes.io/projected/f952ba2e-5049-4a57-851c-7ee424a7b75b-kube-api-access-zq76b\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.172614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.172587 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3692a26a-a911-44ea-b9fa-3c91af5d7748" path="/var/lib/kubelet/pods/3692a26a-a911-44ea-b9fa-3c91af5d7748/volumes" Nov 03 10:50:12.172988 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.172974 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9feef9dd-d6c6-4cd0-8928-259d28809ce1" path="/var/lib/kubelet/pods/9feef9dd-d6c6-4cd0-8928-259d28809ce1/volumes" Nov 03 10:50:12.625852 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.625813 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.628830 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.628800 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:12.831075 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:12.831040 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:13.114536 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:13.114505 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr"] Nov 03 10:50:13.116020 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:50:13.115991 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf952ba2e_5049_4a57_851c_7ee424a7b75b.slice/crio-0d14c29fac7cdd517c0704e316a58537a8a1302611f814af7ce3f86535da6955 WatchSource:0}: Error finding container 0d14c29fac7cdd517c0704e316a58537a8a1302611f814af7ce3f86535da6955: Status 404 returned error can't find the container with id 0d14c29fac7cdd517c0704e316a58537a8a1302611f814af7ce3f86535da6955 Nov 03 10:50:13.915387 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:13.915332 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" event={"ID":"f952ba2e-5049-4a57-851c-7ee424a7b75b","Type":"ContainerStarted","Data":"527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e"} Nov 03 10:50:13.915800 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:13.915411 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" event={"ID":"f952ba2e-5049-4a57-851c-7ee424a7b75b","Type":"ContainerStarted","Data":"0d14c29fac7cdd517c0704e316a58537a8a1302611f814af7ce3f86535da6955"} Nov 03 10:50:13.915800 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:13.915533 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:13.936322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:13.936254 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" podStartSLOduration=2.502823146 podStartE2EDuration="2.93624048s" totalImagesPullingTime="433.417334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:13.932747957 +0000 UTC m=+1291.917619262" watchObservedRunningTime="2026-11-03 10:50:13.93624048 +0000 UTC m=+1291.921111762" Nov 03 10:50:15.840520 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:15.840489 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:50:15.841012 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:15.840892 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:50:19.939208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:19.939179 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:50:24.280734 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:50:24.280653 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:50:32.663032 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:32.662986 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh"] Nov 03 10:50:32.663590 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:32.663244 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" podUID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerName="maas-api" containerID="cri-o://1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792" gracePeriod=30 Nov 03 10:50:32.993914 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:32.993577 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:50:33.095896 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.095854 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-metrics-certs\") pod \"ef6d124d-4f64-41fc-ad08-a3878a62da29\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " Nov 03 10:50:33.096688 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.096666 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-maas-api-tls\") pod \"ef6d124d-4f64-41fc-ad08-a3878a62da29\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " Nov 03 10:50:33.097263 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.097241 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6d124d-4f64-41fc-ad08-a3878a62da29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6d124d-4f64-41fc-ad08-a3878a62da29-openshift-service-ca\") pod \"ef6d124d-4f64-41fc-ad08-a3878a62da29\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " Nov 03 10:50:33.099459 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.097752 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6d124d-4f64-41fc-ad08-a3878a62da29-openshift-service-ca" pod "ef6d124d-4f64-41fc-ad08-a3878a62da29" (UID: "ef6d124d-4f64-41fc-ad08-a3878a62da29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:50:33.099560 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.099436 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-metrics-certs" pod "ef6d124d-4f64-41fc-ad08-a3878a62da29" (UID: "ef6d124d-4f64-41fc-ad08-a3878a62da29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:50:33.099560 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.099485 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6d124d-4f64-41fc-ad08-a3878a62da29-kube-api-access-h6xfv\" (UniqueName: \"kubernetes.io/projected/ef6d124d-4f64-41fc-ad08-a3878a62da29-kube-api-access-h6xfv\") pod \"ef6d124d-4f64-41fc-ad08-a3878a62da29\" (UID: \"ef6d124d-4f64-41fc-ad08-a3878a62da29\") " Nov 03 10:50:33.099560 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.099496 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-maas-api-tls" pod "ef6d124d-4f64-41fc-ad08-a3878a62da29" (UID: "ef6d124d-4f64-41fc-ad08-a3878a62da29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:50:33.100166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.100134 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:33.100166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.100159 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6d124d-4f64-41fc-ad08-a3878a62da29-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:33.100370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.100173 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6d124d-4f64-41fc-ad08-a3878a62da29-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:33.101829 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.101807 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6d124d-4f64-41fc-ad08-a3878a62da29-kube-api-access-h6xfv" pod "ef6d124d-4f64-41fc-ad08-a3878a62da29" (UID: "ef6d124d-4f64-41fc-ad08-a3878a62da29"). InnerVolumeSpecName "kube-api-access-h6xfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:50:33.138049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.138010 5086 generic.go:356] "Generic (PLEG): container finished" podID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerID="1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792" exitCode=0 Nov 03 10:50:33.138252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.138054 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" event={"ID":"ef6d124d-4f64-41fc-ad08-a3878a62da29","Type":"ContainerDied","Data":"1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792"} Nov 03 10:50:33.138252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.138073 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" Nov 03 10:50:33.138252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.138123 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh" event={"ID":"ef6d124d-4f64-41fc-ad08-a3878a62da29","Type":"ContainerDied","Data":"5e8f3c9e9e25d7687e8cb847a7f4c9ef10370a92ed18456bc0e4f32fcf1c73e4"} Nov 03 10:50:33.138252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.138154 5086 scope.go:122] "RemoveContainer" containerID="1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792" Nov 03 10:50:33.176198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.150917 5086 scope.go:122] "RemoveContainer" containerID="9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756" Nov 03 10:50:33.176198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.172458 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh"] Nov 03 10:50:33.176198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.174517 5086 scope.go:122] "RemoveContainer" containerID="1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792" Nov 03 10:50:33.176198 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:50:33.174898 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792\": container with ID starting with 1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792 not found: ID does not exist" containerID="1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792" Nov 03 10:50:33.176198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.174924 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792"} err="failed to get container status \"1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792\": rpc error: code = NotFound desc = could not find container \"1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792\": container with ID starting with 1e52e62865ec7ecd0adcf1a9d2d3fb3e86b0058f53169b421e82d059f6119792 not found: ID does not exist" Nov 03 10:50:33.176198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.174943 5086 scope.go:122] "RemoveContainer" containerID="9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756" Nov 03 10:50:33.176198 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:50:33.175255 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756\": container with ID starting with 9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756 not found: ID does not exist" containerID="9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756" Nov 03 10:50:33.176198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.175286 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756"} err="failed to get container status \"9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756\": rpc error: code = NotFound desc = could not find container \"9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756\": container with ID starting with 9c2092d2cc72d9f215b3d63e310ce780e9d137e750d42f0cc81eafecff208756 not found: ID does not exist" Nov 03 10:50:33.176198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.176136 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38c9b0-5c988bcb55-mbzsh"] Nov 03 10:50:33.201651 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:33.201608 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6xfv\" (UniqueName: \"kubernetes.io/projected/ef6d124d-4f64-41fc-ad08-a3878a62da29-kube-api-access-h6xfv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:50:34.187823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:34.187789 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6d124d-4f64-41fc-ad08-a3878a62da29" path="/var/lib/kubelet/pods/ef6d124d-4f64-41fc-ad08-a3878a62da29/volumes" Nov 03 10:50:36.279504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.279467 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs"] Nov 03 10:50:36.280590 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.280569 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerName="maas-api" Nov 03 10:50:36.280646 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.280594 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerName="maas-api" Nov 03 10:50:36.280646 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.280617 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerName="maas-api" Nov 03 10:50:36.280646 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.280625 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerName="maas-api" Nov 03 10:50:36.280874 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.280859 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerName="maas-api" Nov 03 10:50:36.280915 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.280882 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6d124d-4f64-41fc-ad08-a3878a62da29" containerName="maas-api" Nov 03 10:50:36.300173 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.300137 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs"] Nov 03 10:50:36.300446 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.300183 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:36.304418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.304390 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a2697c-dockercfg-nsl5p\"" Nov 03 10:50:36.345725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.345693 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84cb267c-9d88-44ea-9178-ae6146181af1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs\" (UID: \"84cb267c-9d88-44ea-9178-ae6146181af1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:36.345894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.345873 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvnhl\" (UniqueName: \"kubernetes.io/projected/84cb267c-9d88-44ea-9178-ae6146181af1-kube-api-access-rvnhl\") pod \"payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs\" (UID: \"84cb267c-9d88-44ea-9178-ae6146181af1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:36.363210 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.363176 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a2697c\"" Nov 03 10:50:36.447435 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.447401 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84cb267c-9d88-44ea-9178-ae6146181af1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs\" (UID: \"84cb267c-9d88-44ea-9178-ae6146181af1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:36.447644 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.447514 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvnhl\" (UniqueName: \"kubernetes.io/projected/84cb267c-9d88-44ea-9178-ae6146181af1-kube-api-access-rvnhl\") pod \"payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs\" (UID: \"84cb267c-9d88-44ea-9178-ae6146181af1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:36.448250 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.448225 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84cb267c-9d88-44ea-9178-ae6146181af1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs\" (UID: \"84cb267c-9d88-44ea-9178-ae6146181af1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:36.455070 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.455044 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvnhl\" (UniqueName: \"kubernetes.io/projected/84cb267c-9d88-44ea-9178-ae6146181af1-kube-api-access-rvnhl\") pod \"payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs\" (UID: \"84cb267c-9d88-44ea-9178-ae6146181af1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:36.620015 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.619978 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:36.783669 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.783635 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469"] Nov 03 10:50:36.794112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.794065 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.799006 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.798979 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469"] Nov 03 10:50:36.801105 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.801077 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-814dc0-serving-cert\"" Nov 03 10:50:36.855505 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.855469 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e705d8fa-f289-456c-a228-768f165311e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.855724 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.855704 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.855811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.855746 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-metrics-certs\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.855877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.855835 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sz59\" (UniqueName: \"kubernetes.io/projected/e705d8fa-f289-456c-a228-768f165311e5-kube-api-access-2sz59\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.911756 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.911727 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs"] Nov 03 10:50:36.913767 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:50:36.913739 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cb267c_9d88_44ea_9178_ae6146181af1.slice/crio-2a3af3c0766204e341f426a498d0d6cfaec7806fd8641ccb8c8f06c75665a449 WatchSource:0}: Error finding container 2a3af3c0766204e341f426a498d0d6cfaec7806fd8641ccb8c8f06c75665a449: Status 404 returned error can't find the container with id 2a3af3c0766204e341f426a498d0d6cfaec7806fd8641ccb8c8f06c75665a449 Nov 03 10:50:36.957694 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.957659 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.957869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.957715 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-metrics-certs\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.957869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.957762 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sz59\" (UniqueName: \"kubernetes.io/projected/e705d8fa-f289-456c-a228-768f165311e5-kube-api-access-2sz59\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.957964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.957914 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e705d8fa-f289-456c-a228-768f165311e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.958956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.958931 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e705d8fa-f289-456c-a228-768f165311e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.961925 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.961902 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-metrics-certs\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.962068 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.961930 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:36.966482 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:36.966459 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sz59\" (UniqueName: \"kubernetes.io/projected/e705d8fa-f289-456c-a228-768f165311e5-kube-api-access-2sz59\") pod \"maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:37.117493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:37.117452 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:37.187182 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:37.184659 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" event={"ID":"84cb267c-9d88-44ea-9178-ae6146181af1","Type":"ContainerStarted","Data":"f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61"} Nov 03 10:50:37.187182 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:37.184731 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" event={"ID":"84cb267c-9d88-44ea-9178-ae6146181af1","Type":"ContainerStarted","Data":"2a3af3c0766204e341f426a498d0d6cfaec7806fd8641ccb8c8f06c75665a449"} Nov 03 10:50:37.187182 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:37.184783 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:37.212122 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:37.211930 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" podStartSLOduration=1.211909932 podStartE2EDuration="1.211909932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:37.206660127 +0000 UTC m=+1315.191531412" watchObservedRunningTime="2026-11-03 10:50:37.211909932 +0000 UTC m=+1315.196781216" Nov 03 10:50:37.423810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:37.423762 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469"] Nov 03 10:50:37.425228 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:50:37.425201 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode705d8fa_f289_456c_a228_768f165311e5.slice/crio-e2c619a55e573a93581ab65fb016eccc5ffbeee94f5ef8f286ed7c1b1ad8351b WatchSource:0}: Error finding container e2c619a55e573a93581ab65fb016eccc5ffbeee94f5ef8f286ed7c1b1ad8351b: Status 404 returned error can't find the container with id e2c619a55e573a93581ab65fb016eccc5ffbeee94f5ef8f286ed7c1b1ad8351b Nov 03 10:50:38.221193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:38.221104 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" event={"ID":"e705d8fa-f289-456c-a228-768f165311e5","Type":"ContainerStarted","Data":"43db04fc76c9f4d9eca799ca538a7099a131786c6a272f7e9d775effef7d5a34"} Nov 03 10:50:38.221536 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:38.221457 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" event={"ID":"e705d8fa-f289-456c-a228-768f165311e5","Type":"ContainerStarted","Data":"e2c619a55e573a93581ab65fb016eccc5ffbeee94f5ef8f286ed7c1b1ad8351b"} Nov 03 10:50:38.221945 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:38.221727 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:38.243455 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:38.243403 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" podStartSLOduration=1.7659167930000002 podStartE2EDuration="2.24338885s" totalImagesPullingTime="477.472057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:38.24237018 +0000 UTC m=+1316.227241486" watchObservedRunningTime="2026-11-03 10:50:38.24338885 +0000 UTC m=+1316.228260132" Nov 03 10:50:45.244535 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:45.244506 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:50:48.223276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:50:48.223244 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:50:54.279008 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:50:54.278959 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:51:05.665144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.665101 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s"] Nov 03 10:51:05.677096 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.673661 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.690421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.683130 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b3dc04-serving-cert\"" Nov 03 10:51:05.690421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.689033 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s"] Nov 03 10:51:05.761002 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.760958 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a09f09-e2e9-4356-b3fb-5860237c80cf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.761219 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.761113 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.761391 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.761371 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86lt7\" (UniqueName: \"kubernetes.io/projected/d8a09f09-e2e9-4356-b3fb-5860237c80cf-kube-api-access-86lt7\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.761524 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.761496 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-metrics-certs\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.863016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.862977 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-metrics-certs\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.863384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.863320 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a09f09-e2e9-4356-b3fb-5860237c80cf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.863628 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.863444 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.864442 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.864414 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a09f09-e2e9-4356-b3fb-5860237c80cf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.865183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.864543 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86lt7\" (UniqueName: \"kubernetes.io/projected/d8a09f09-e2e9-4356-b3fb-5860237c80cf-kube-api-access-86lt7\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.876019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.871564 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.876019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.874255 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-metrics-certs\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.877129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.877041 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86lt7\" (UniqueName: \"kubernetes.io/projected/d8a09f09-e2e9-4356-b3fb-5860237c80cf-kube-api-access-86lt7\") pod \"maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:05.999857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:05.999765 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:06.309276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:06.309190 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s"] Nov 03 10:51:06.316736 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:51:06.316694 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a09f09_e2e9_4356_b3fb_5860237c80cf.slice/crio-9dba72543a00c2329248d8fd05f1a2b57e1f3d0de7e6660fc52fb421b4f40fb0 WatchSource:0}: Error finding container 9dba72543a00c2329248d8fd05f1a2b57e1f3d0de7e6660fc52fb421b4f40fb0: Status 404 returned error can't find the container with id 9dba72543a00c2329248d8fd05f1a2b57e1f3d0de7e6660fc52fb421b4f40fb0 Nov 03 10:51:06.318516 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:06.318496 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:51:06.464243 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:06.464199 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" event={"ID":"d8a09f09-e2e9-4356-b3fb-5860237c80cf","Type":"ContainerStarted","Data":"9dba72543a00c2329248d8fd05f1a2b57e1f3d0de7e6660fc52fb421b4f40fb0"} Nov 03 10:51:07.474749 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:07.474709 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" event={"ID":"d8a09f09-e2e9-4356-b3fb-5860237c80cf","Type":"ContainerStarted","Data":"46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554"} Nov 03 10:51:07.475264 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:07.474803 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:07.492254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:07.492198 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" podStartSLOduration=2.062748013 podStartE2EDuration="2.492181553s" totalImagesPullingTime="429.43354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:07.491694943 +0000 UTC m=+1345.476566226" watchObservedRunningTime="2026-11-03 10:51:07.492181553 +0000 UTC m=+1345.477052837" Nov 03 10:51:10.216492 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.216456 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc"] Nov 03 10:51:10.216971 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.216752 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" podUID="d1e1019a-4d07-44ed-935e-67d588004dd1" containerName="maas-api" containerID="cri-o://1230d4ad195318f3834c152b96a32386dd9294d591a2f91996cffb3f7a25f507" gracePeriod=30 Nov 03 10:51:10.389208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.378753 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls"] Nov 03 10:51:10.389208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.379027 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" podUID="3ebe5575-ee5c-448e-8858-0c6e3803efe1" containerName="payload-processing" containerID="cri-o://45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076" gracePeriod=30 Nov 03 10:51:10.389208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.389047 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw"] Nov 03 10:51:10.389617 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.389265 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" podUID="046018fb-6293-435f-a62e-9d1fccde20b3" containerName="payload-pre-processing" containerID="cri-o://6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83" gracePeriod=30 Nov 03 10:51:10.515474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.513561 5086 generic.go:356] "Generic (PLEG): container finished" podID="d1e1019a-4d07-44ed-935e-67d588004dd1" containerID="1230d4ad195318f3834c152b96a32386dd9294d591a2f91996cffb3f7a25f507" exitCode=0 Nov 03 10:51:10.515474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.513628 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" event={"ID":"d1e1019a-4d07-44ed-935e-67d588004dd1","Type":"ContainerDied","Data":"1230d4ad195318f3834c152b96a32386dd9294d591a2f91996cffb3f7a25f507"} Nov 03 10:51:10.605434 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:51:10.605385 5086 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebe5575_ee5c_448e_8858_0c6e3803efe1.slice/crio-45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebe5575_ee5c_448e_8858_0c6e3803efe1.slice/crio-conmon-45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076.scope\": RecentStats: unable to find data in memory cache]" Nov 03 10:51:10.647849 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.647816 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:51:10.736342 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.736254 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-maas-api-tls\") pod \"d1e1019a-4d07-44ed-935e-67d588004dd1\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " Nov 03 10:51:10.736590 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.736568 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e1019a-4d07-44ed-935e-67d588004dd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e1019a-4d07-44ed-935e-67d588004dd1-openshift-service-ca\") pod \"d1e1019a-4d07-44ed-935e-67d588004dd1\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " Nov 03 10:51:10.736685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.736652 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e1019a-4d07-44ed-935e-67d588004dd1-kube-api-access-xlpfd\" (UniqueName: \"kubernetes.io/projected/d1e1019a-4d07-44ed-935e-67d588004dd1-kube-api-access-xlpfd\") pod \"d1e1019a-4d07-44ed-935e-67d588004dd1\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " Nov 03 10:51:10.736753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.736691 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-metrics-certs\") pod \"d1e1019a-4d07-44ed-935e-67d588004dd1\" (UID: \"d1e1019a-4d07-44ed-935e-67d588004dd1\") " Nov 03 10:51:10.737039 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.737007 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e1019a-4d07-44ed-935e-67d588004dd1-openshift-service-ca" pod "d1e1019a-4d07-44ed-935e-67d588004dd1" (UID: "d1e1019a-4d07-44ed-935e-67d588004dd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:51:10.737408 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.737382 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e1019a-4d07-44ed-935e-67d588004dd1-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:10.741340 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.739609 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-maas-api-tls" pod "d1e1019a-4d07-44ed-935e-67d588004dd1" (UID: "d1e1019a-4d07-44ed-935e-67d588004dd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:51:10.741340 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.739945 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e1019a-4d07-44ed-935e-67d588004dd1-kube-api-access-xlpfd" pod "d1e1019a-4d07-44ed-935e-67d588004dd1" (UID: "d1e1019a-4d07-44ed-935e-67d588004dd1"). InnerVolumeSpecName "kube-api-access-xlpfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:51:10.741340 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.740131 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-metrics-certs" pod "d1e1019a-4d07-44ed-935e-67d588004dd1" (UID: "d1e1019a-4d07-44ed-935e-67d588004dd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:51:10.769974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.769950 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:51:10.816683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.816650 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:51:10.910024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.906783 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046018fb-6293-435f-a62e-9d1fccde20b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046018fb-6293-435f-a62e-9d1fccde20b3-plugins-config-volume\") pod \"046018fb-6293-435f-a62e-9d1fccde20b3\" (UID: \"046018fb-6293-435f-a62e-9d1fccde20b3\") " Nov 03 10:51:10.910024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.906871 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046018fb-6293-435f-a62e-9d1fccde20b3-kube-api-access-rrzk9\" (UniqueName: \"kubernetes.io/projected/046018fb-6293-435f-a62e-9d1fccde20b3-kube-api-access-rrzk9\") pod \"046018fb-6293-435f-a62e-9d1fccde20b3\" (UID: \"046018fb-6293-435f-a62e-9d1fccde20b3\") " Nov 03 10:51:10.910024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.907857 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlpfd\" (UniqueName: \"kubernetes.io/projected/d1e1019a-4d07-44ed-935e-67d588004dd1-kube-api-access-xlpfd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:10.910024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.907886 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:10.910024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.907901 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e1019a-4d07-44ed-935e-67d588004dd1-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:10.924349 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.923951 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046018fb-6293-435f-a62e-9d1fccde20b3-kube-api-access-rrzk9" pod "046018fb-6293-435f-a62e-9d1fccde20b3" (UID: "046018fb-6293-435f-a62e-9d1fccde20b3"). InnerVolumeSpecName "kube-api-access-rrzk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:51:10.951534 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:10.951448 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046018fb-6293-435f-a62e-9d1fccde20b3-plugins-config-volume" pod "046018fb-6293-435f-a62e-9d1fccde20b3" (UID: "046018fb-6293-435f-a62e-9d1fccde20b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:51:11.009791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.009688 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ebe5575-ee5c-448e-8858-0c6e3803efe1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebe5575-ee5c-448e-8858-0c6e3803efe1-plugins-config-volume\") pod \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\" (UID: \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\") " Nov 03 10:51:11.009974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.009844 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ebe5575-ee5c-448e-8858-0c6e3803efe1-kube-api-access-6txdc\" (UniqueName: \"kubernetes.io/projected/3ebe5575-ee5c-448e-8858-0c6e3803efe1-kube-api-access-6txdc\") pod \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\" (UID: \"3ebe5575-ee5c-448e-8858-0c6e3803efe1\") " Nov 03 10:51:11.010099 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.010075 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebe5575-ee5c-448e-8858-0c6e3803efe1-plugins-config-volume" pod "3ebe5575-ee5c-448e-8858-0c6e3803efe1" (UID: "3ebe5575-ee5c-448e-8858-0c6e3803efe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:51:11.010410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.010388 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ebe5575-ee5c-448e-8858-0c6e3803efe1-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:11.010410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.010407 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046018fb-6293-435f-a62e-9d1fccde20b3-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:11.010569 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.010421 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrzk9\" (UniqueName: \"kubernetes.io/projected/046018fb-6293-435f-a62e-9d1fccde20b3-kube-api-access-rrzk9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:11.012077 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.012044 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebe5575-ee5c-448e-8858-0c6e3803efe1-kube-api-access-6txdc" pod "3ebe5575-ee5c-448e-8858-0c6e3803efe1" (UID: "3ebe5575-ee5c-448e-8858-0c6e3803efe1"). InnerVolumeSpecName "kube-api-access-6txdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:51:11.111446 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.111411 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6txdc\" (UniqueName: \"kubernetes.io/projected/3ebe5575-ee5c-448e-8858-0c6e3803efe1-kube-api-access-6txdc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:11.522253 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.522218 5086 generic.go:356] "Generic (PLEG): container finished" podID="046018fb-6293-435f-a62e-9d1fccde20b3" containerID="6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83" exitCode=0 Nov 03 10:51:11.522823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.522263 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" Nov 03 10:51:11.522823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.522344 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" event={"ID":"046018fb-6293-435f-a62e-9d1fccde20b3","Type":"ContainerDied","Data":"6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83"} Nov 03 10:51:11.522823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.522412 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw" event={"ID":"046018fb-6293-435f-a62e-9d1fccde20b3","Type":"ContainerDied","Data":"a1d2f7ab1ee7f919615891e0dbee5b6722050acfbab408e8ca06450d54ca608b"} Nov 03 10:51:11.522823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.522445 5086 scope.go:122] "RemoveContainer" containerID="6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83" Nov 03 10:51:11.524489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.524465 5086 generic.go:356] "Generic (PLEG): container finished" podID="3ebe5575-ee5c-448e-8858-0c6e3803efe1" containerID="45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076" exitCode=0 Nov 03 10:51:11.524606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.524515 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" Nov 03 10:51:11.524606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.524545 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" event={"ID":"3ebe5575-ee5c-448e-8858-0c6e3803efe1","Type":"ContainerDied","Data":"45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076"} Nov 03 10:51:11.524606 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.524603 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls" event={"ID":"3ebe5575-ee5c-448e-8858-0c6e3803efe1","Type":"ContainerDied","Data":"b89dd5259f83bb1db7dbfbaa81c1a8d324c019238ec8ab42995615a9258bf9de"} Nov 03 10:51:11.526068 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.526047 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" event={"ID":"d1e1019a-4d07-44ed-935e-67d588004dd1","Type":"ContainerDied","Data":"33ecd89c0aa1443a7c60714e7ad00d5883cd6e0583981126aacb4031d4f0e55d"} Nov 03 10:51:11.526179 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.526096 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc" Nov 03 10:51:11.534602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.534586 5086 scope.go:122] "RemoveContainer" containerID="6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83" Nov 03 10:51:11.534883 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:51:11.534863 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83\": container with ID starting with 6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83 not found: ID does not exist" containerID="6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83" Nov 03 10:51:11.534978 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.534887 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83"} err="failed to get container status \"6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83\": rpc error: code = NotFound desc = could not find container \"6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83\": container with ID starting with 6aafc66ea7c5f094d0a36ee5ea8a48b282f3c74db758c5c691af1c91a0e36c83 not found: ID does not exist" Nov 03 10:51:11.534978 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.534905 5086 scope.go:122] "RemoveContainer" containerID="45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076" Nov 03 10:51:11.551201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.551172 5086 scope.go:122] "RemoveContainer" containerID="45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076" Nov 03 10:51:11.551625 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:51:11.551593 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076\": container with ID starting with 45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076 not found: ID does not exist" containerID="45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076" Nov 03 10:51:11.551770 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.551636 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076"} err="failed to get container status \"45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076\": rpc error: code = NotFound desc = could not find container \"45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076\": container with ID starting with 45ac36e59adb700a612a48dfd81844937989fee6780febed19a525636ea77076 not found: ID does not exist" Nov 03 10:51:11.551770 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.551662 5086 scope.go:122] "RemoveContainer" containerID="1230d4ad195318f3834c152b96a32386dd9294d591a2f91996cffb3f7a25f507" Nov 03 10:51:11.563904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.563881 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc"] Nov 03 10:51:11.567588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.567565 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de30a0-b47797c65-4qrbc"] Nov 03 10:51:11.579123 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.579098 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw"] Nov 03 10:51:11.582632 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.582609 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de30a0-64dd64855f-s6gjw"] Nov 03 10:51:11.592411 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.592383 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls"] Nov 03 10:51:11.595711 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:11.595691 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de30a0-5c7f9478bd-8sqls"] Nov 03 10:51:12.207322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:12.198699 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046018fb-6293-435f-a62e-9d1fccde20b3" path="/var/lib/kubelet/pods/046018fb-6293-435f-a62e-9d1fccde20b3/volumes" Nov 03 10:51:12.207322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:12.200963 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ebe5575-ee5c-448e-8858-0c6e3803efe1" path="/var/lib/kubelet/pods/3ebe5575-ee5c-448e-8858-0c6e3803efe1/volumes" Nov 03 10:51:12.207322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:12.201675 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e1019a-4d07-44ed-935e-67d588004dd1" path="/var/lib/kubelet/pods/d1e1019a-4d07-44ed-935e-67d588004dd1/volumes" Nov 03 10:51:13.483444 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:13.483399 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" podUID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.139:8443/health\": dial tcp 10.132.0.139:8443: connect: connection refused" Nov 03 10:51:17.616317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:17.615396 5086 generic.go:356] "Generic (PLEG): container finished" podID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerID="46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554" exitCode=1 Nov 03 10:51:17.616317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:17.615666 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" event={"ID":"d8a09f09-e2e9-4356-b3fb-5860237c80cf","Type":"ContainerDied","Data":"46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554"} Nov 03 10:51:17.616317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:17.616216 5086 scope.go:122] "RemoveContainer" containerID="46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554" Nov 03 10:51:18.624681 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:18.624644 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" event={"ID":"d8a09f09-e2e9-4356-b3fb-5860237c80cf","Type":"ContainerStarted","Data":"0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02"} Nov 03 10:51:18.625188 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:18.625052 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:24.406241 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:51:24.406196 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:51:24.638392 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:24.638361 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:51:28.941395 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:28.941357 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr"] Nov 03 10:51:28.941991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:28.941962 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" podUID="f952ba2e-5049-4a57-851c-7ee424a7b75b" containerName="maas-api" containerID="cri-o://527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e" gracePeriod=30 Nov 03 10:51:29.013098 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.013059 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs"] Nov 03 10:51:29.013462 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.013432 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" podUID="84cb267c-9d88-44ea-9178-ae6146181af1" containerName="payload-processing" containerID="cri-o://f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61" gracePeriod=30 Nov 03 10:51:29.388784 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.388754 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:51:29.469048 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.469022 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:51:29.490826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.490777 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls\") pod \"f952ba2e-5049-4a57-851c-7ee424a7b75b\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " Nov 03 10:51:29.491129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.491110 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f952ba2e-5049-4a57-851c-7ee424a7b75b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f952ba2e-5049-4a57-851c-7ee424a7b75b-openshift-service-ca\") pod \"f952ba2e-5049-4a57-851c-7ee424a7b75b\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " Nov 03 10:51:29.491245 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.491209 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-metrics-certs\") pod \"f952ba2e-5049-4a57-851c-7ee424a7b75b\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " Nov 03 10:51:29.491545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.491512 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f952ba2e-5049-4a57-851c-7ee424a7b75b-openshift-service-ca" pod "f952ba2e-5049-4a57-851c-7ee424a7b75b" (UID: "f952ba2e-5049-4a57-851c-7ee424a7b75b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:51:29.491778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.491755 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f952ba2e-5049-4a57-851c-7ee424a7b75b-kube-api-access-zq76b\" (UniqueName: \"kubernetes.io/projected/f952ba2e-5049-4a57-851c-7ee424a7b75b-kube-api-access-zq76b\") pod \"f952ba2e-5049-4a57-851c-7ee424a7b75b\" (UID: \"f952ba2e-5049-4a57-851c-7ee424a7b75b\") " Nov 03 10:51:29.492410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.492386 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f952ba2e-5049-4a57-851c-7ee424a7b75b-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:29.493823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.493795 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-metrics-certs" pod "f952ba2e-5049-4a57-851c-7ee424a7b75b" (UID: "f952ba2e-5049-4a57-851c-7ee424a7b75b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:51:29.493975 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.493954 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls" pod "f952ba2e-5049-4a57-851c-7ee424a7b75b" (UID: "f952ba2e-5049-4a57-851c-7ee424a7b75b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:51:29.494759 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.494736 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f952ba2e-5049-4a57-851c-7ee424a7b75b-kube-api-access-zq76b" pod "f952ba2e-5049-4a57-851c-7ee424a7b75b" (UID: "f952ba2e-5049-4a57-851c-7ee424a7b75b"). InnerVolumeSpecName "kube-api-access-zq76b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:51:29.593551 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.593511 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84cb267c-9d88-44ea-9178-ae6146181af1-kube-api-access-rvnhl\" (UniqueName: \"kubernetes.io/projected/84cb267c-9d88-44ea-9178-ae6146181af1-kube-api-access-rvnhl\") pod \"84cb267c-9d88-44ea-9178-ae6146181af1\" (UID: \"84cb267c-9d88-44ea-9178-ae6146181af1\") " Nov 03 10:51:29.593736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.593584 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84cb267c-9d88-44ea-9178-ae6146181af1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84cb267c-9d88-44ea-9178-ae6146181af1-plugins-config-volume\") pod \"84cb267c-9d88-44ea-9178-ae6146181af1\" (UID: \"84cb267c-9d88-44ea-9178-ae6146181af1\") " Nov 03 10:51:29.594001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.593969 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84cb267c-9d88-44ea-9178-ae6146181af1-plugins-config-volume" pod "84cb267c-9d88-44ea-9178-ae6146181af1" (UID: "84cb267c-9d88-44ea-9178-ae6146181af1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:51:29.594402 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.594383 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq76b\" (UniqueName: \"kubernetes.io/projected/f952ba2e-5049-4a57-851c-7ee424a7b75b-kube-api-access-zq76b\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:29.594464 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.594410 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:29.594464 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.594425 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84cb267c-9d88-44ea-9178-ae6146181af1-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:29.594464 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.594439 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f952ba2e-5049-4a57-851c-7ee424a7b75b-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:29.595949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.595924 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84cb267c-9d88-44ea-9178-ae6146181af1-kube-api-access-rvnhl" pod "84cb267c-9d88-44ea-9178-ae6146181af1" (UID: "84cb267c-9d88-44ea-9178-ae6146181af1"). InnerVolumeSpecName "kube-api-access-rvnhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:51:29.696122 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.696082 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvnhl\" (UniqueName: \"kubernetes.io/projected/84cb267c-9d88-44ea-9178-ae6146181af1-kube-api-access-rvnhl\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:29.705833 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.705798 5086 generic.go:356] "Generic (PLEG): container finished" podID="84cb267c-9d88-44ea-9178-ae6146181af1" containerID="f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61" exitCode=0 Nov 03 10:51:29.706037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.705861 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" Nov 03 10:51:29.706037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.705881 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" event={"ID":"84cb267c-9d88-44ea-9178-ae6146181af1","Type":"ContainerDied","Data":"f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61"} Nov 03 10:51:29.706037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.705966 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs" event={"ID":"84cb267c-9d88-44ea-9178-ae6146181af1","Type":"ContainerDied","Data":"2a3af3c0766204e341f426a498d0d6cfaec7806fd8641ccb8c8f06c75665a449"} Nov 03 10:51:29.706037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.705991 5086 scope.go:122] "RemoveContainer" containerID="f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61" Nov 03 10:51:29.708930 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.708775 5086 generic.go:356] "Generic (PLEG): container finished" podID="f952ba2e-5049-4a57-851c-7ee424a7b75b" containerID="527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e" exitCode=0 Nov 03 10:51:29.708930 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.708846 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" event={"ID":"f952ba2e-5049-4a57-851c-7ee424a7b75b","Type":"ContainerDied","Data":"527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e"} Nov 03 10:51:29.708930 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.708892 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" event={"ID":"f952ba2e-5049-4a57-851c-7ee424a7b75b","Type":"ContainerDied","Data":"0d14c29fac7cdd517c0704e316a58537a8a1302611f814af7ce3f86535da6955"} Nov 03 10:51:29.708930 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.708896 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr" Nov 03 10:51:29.721803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.721778 5086 scope.go:122] "RemoveContainer" containerID="f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61" Nov 03 10:51:29.722147 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:51:29.722127 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61\": container with ID starting with f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61 not found: ID does not exist" containerID="f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61" Nov 03 10:51:29.722197 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.722157 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61"} err="failed to get container status \"f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61\": rpc error: code = NotFound desc = could not find container \"f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61\": container with ID starting with f35c4c91584470141d64c3f8e03c674ea371375134a100a4d60bf6247aa62a61 not found: ID does not exist" Nov 03 10:51:29.722197 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.722174 5086 scope.go:122] "RemoveContainer" containerID="527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e" Nov 03 10:51:29.733391 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.733363 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs"] Nov 03 10:51:29.735013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.734998 5086 scope.go:122] "RemoveContainer" containerID="527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e" Nov 03 10:51:29.735394 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:51:29.735367 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e\": container with ID starting with 527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e not found: ID does not exist" containerID="527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e" Nov 03 10:51:29.735496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.735404 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e"} err="failed to get container status \"527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e\": rpc error: code = NotFound desc = could not find container \"527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e\": container with ID starting with 527c19a8fc72bc0b1ec498c77d7effd22333c9fcdd8e0bd9a487fa391120d52e not found: ID does not exist" Nov 03 10:51:29.736971 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.736950 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-777dbd5db8-5jccs"] Nov 03 10:51:29.749665 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.749637 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr"] Nov 03 10:51:29.753036 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:29.753011 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2697c-59d5cbc7c5-kwgsr"] Nov 03 10:51:30.172095 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:30.172065 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84cb267c-9d88-44ea-9178-ae6146181af1" path="/var/lib/kubelet/pods/84cb267c-9d88-44ea-9178-ae6146181af1/volumes" Nov 03 10:51:30.172586 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:30.172570 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f952ba2e-5049-4a57-851c-7ee424a7b75b" path="/var/lib/kubelet/pods/f952ba2e-5049-4a57-851c-7ee424a7b75b/volumes" Nov 03 10:51:32.438249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.435586 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf"] Nov 03 10:51:32.438249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.435867 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" podUID="f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" containerName="maas-api" containerID="cri-o://9b4314073d3fe909bcf9f8420ad9610a8ffdad5f2ca4e08d13af8731fed5b2a8" gracePeriod=30 Nov 03 10:51:32.517015 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.512192 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5"] Nov 03 10:51:32.517015 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.512481 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" podUID="f21e89d4-3943-4edb-ad24-4718c7f19ed1" containerName="payload-pre-processing" containerID="cri-o://62244394fec9d83711f0f1e857299bdb3fc80736e465f305f8f228ae2986c712" gracePeriod=30 Nov 03 10:51:32.778365 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.778073 5086 generic.go:356] "Generic (PLEG): container finished" podID="f21e89d4-3943-4edb-ad24-4718c7f19ed1" containerID="62244394fec9d83711f0f1e857299bdb3fc80736e465f305f8f228ae2986c712" exitCode=0 Nov 03 10:51:32.778548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.778499 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" event={"ID":"f21e89d4-3943-4edb-ad24-4718c7f19ed1","Type":"ContainerDied","Data":"62244394fec9d83711f0f1e857299bdb3fc80736e465f305f8f228ae2986c712"} Nov 03 10:51:32.783587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.781258 5086 generic.go:356] "Generic (PLEG): container finished" podID="f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" containerID="9b4314073d3fe909bcf9f8420ad9610a8ffdad5f2ca4e08d13af8731fed5b2a8" exitCode=0 Nov 03 10:51:32.783587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.781361 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" event={"ID":"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916","Type":"ContainerDied","Data":"9b4314073d3fe909bcf9f8420ad9610a8ffdad5f2ca4e08d13af8731fed5b2a8"} Nov 03 10:51:32.850691 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.849214 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:51:32.959515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.959486 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:51:32.972395 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.972362 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-openshift-service-ca\") pod \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " Nov 03 10:51:32.972570 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.972441 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-maas-api-tls\") pod \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " Nov 03 10:51:32.972570 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.972523 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-metrics-certs\") pod \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " Nov 03 10:51:32.972804 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.972779 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-openshift-service-ca" pod "f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" (UID: "f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:51:32.972879 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.972788 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-kube-api-access-w8bth\" (UniqueName: \"kubernetes.io/projected/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-kube-api-access-w8bth\") pod \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\" (UID: \"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916\") " Nov 03 10:51:32.973568 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.973540 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:32.975197 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.975168 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-maas-api-tls" pod "f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" (UID: "f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:51:32.975370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.975203 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-metrics-certs" pod "f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" (UID: "f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:51:32.975370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:32.975237 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-kube-api-access-w8bth" pod "f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" (UID: "f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916"). InnerVolumeSpecName "kube-api-access-w8bth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:51:33.074591 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.074555 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21e89d4-3943-4edb-ad24-4718c7f19ed1-kube-api-access-69tmc\" (UniqueName: \"kubernetes.io/projected/f21e89d4-3943-4edb-ad24-4718c7f19ed1-kube-api-access-69tmc\") pod \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\" (UID: \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\") " Nov 03 10:51:33.074810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.074679 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21e89d4-3943-4edb-ad24-4718c7f19ed1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21e89d4-3943-4edb-ad24-4718c7f19ed1-plugins-config-volume\") pod \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\" (UID: \"f21e89d4-3943-4edb-ad24-4718c7f19ed1\") " Nov 03 10:51:33.075976 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.075201 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21e89d4-3943-4edb-ad24-4718c7f19ed1-plugins-config-volume" pod "f21e89d4-3943-4edb-ad24-4718c7f19ed1" (UID: "f21e89d4-3943-4edb-ad24-4718c7f19ed1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:51:33.075976 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.075569 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8bth\" (UniqueName: \"kubernetes.io/projected/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-kube-api-access-w8bth\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:33.075976 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.075592 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:33.075976 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.075606 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:33.075976 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.075620 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21e89d4-3943-4edb-ad24-4718c7f19ed1-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:33.079687 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.079658 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21e89d4-3943-4edb-ad24-4718c7f19ed1-kube-api-access-69tmc" pod "f21e89d4-3943-4edb-ad24-4718c7f19ed1" (UID: "f21e89d4-3943-4edb-ad24-4718c7f19ed1"). InnerVolumeSpecName "kube-api-access-69tmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:51:33.176493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.176462 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69tmc\" (UniqueName: \"kubernetes.io/projected/f21e89d4-3943-4edb-ad24-4718c7f19ed1-kube-api-access-69tmc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:51:33.791960 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.791865 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" Nov 03 10:51:33.792432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.791855 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5" event={"ID":"f21e89d4-3943-4edb-ad24-4718c7f19ed1","Type":"ContainerDied","Data":"3c9a4c6465c1be03f91df959e2e21ed8bd5f9519a67178d43ff44981249a368e"} Nov 03 10:51:33.792432 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.792023 5086 scope.go:122] "RemoveContainer" containerID="62244394fec9d83711f0f1e857299bdb3fc80736e465f305f8f228ae2986c712" Nov 03 10:51:33.794787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.794607 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" event={"ID":"f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916","Type":"ContainerDied","Data":"09d49a603c8a35b1e965bdb6f1f991ed0069bd2a75f8d95a698ef54ef8c325fd"} Nov 03 10:51:33.794787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.794692 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf" Nov 03 10:51:33.808802 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.808677 5086 scope.go:122] "RemoveContainer" containerID="9b4314073d3fe909bcf9f8420ad9610a8ffdad5f2ca4e08d13af8731fed5b2a8" Nov 03 10:51:33.837166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.835634 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5"] Nov 03 10:51:33.841765 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.841732 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d9549-99595fb75-4t7j5"] Nov 03 10:51:33.847385 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.847358 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf"] Nov 03 10:51:33.850766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:33.850734 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4d9549-67dc5bf565-z54pf"] Nov 03 10:51:34.171949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:34.171918 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" path="/var/lib/kubelet/pods/f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916/volumes" Nov 03 10:51:34.172393 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:34.172379 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21e89d4-3943-4edb-ad24-4718c7f19ed1" path="/var/lib/kubelet/pods/f21e89d4-3943-4edb-ad24-4718c7f19ed1/volumes" Nov 03 10:51:37.721184 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721154 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm"] Nov 03 10:51:37.721863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721846 5086 cpu_manager.go:414] "containerMap: removing container" podUID="f952ba2e-5049-4a57-851c-7ee424a7b75b" containerName="maas-api" Nov 03 10:51:37.721863 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721863 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f952ba2e-5049-4a57-851c-7ee424a7b75b" containerName="maas-api" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721875 5086 cpu_manager.go:414] "containerMap: removing container" podUID="f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" containerName="maas-api" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721881 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" containerName="maas-api" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721898 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d1e1019a-4d07-44ed-935e-67d588004dd1" containerName="maas-api" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721903 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e1019a-4d07-44ed-935e-67d588004dd1" containerName="maas-api" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721912 5086 cpu_manager.go:414] "containerMap: removing container" podUID="84cb267c-9d88-44ea-9178-ae6146181af1" containerName="payload-processing" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721917 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84cb267c-9d88-44ea-9178-ae6146181af1" containerName="payload-processing" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721930 5086 cpu_manager.go:414] "containerMap: removing container" podUID="3ebe5575-ee5c-448e-8858-0c6e3803efe1" containerName="payload-processing" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721935 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ebe5575-ee5c-448e-8858-0c6e3803efe1" containerName="payload-processing" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721945 5086 cpu_manager.go:414] "containerMap: removing container" podUID="f21e89d4-3943-4edb-ad24-4718c7f19ed1" containerName="payload-pre-processing" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721949 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21e89d4-3943-4edb-ad24-4718c7f19ed1" containerName="payload-pre-processing" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721970 5086 cpu_manager.go:414] "containerMap: removing container" podUID="046018fb-6293-435f-a62e-9d1fccde20b3" containerName="payload-pre-processing" Nov 03 10:51:37.722011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.721974 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046018fb-6293-435f-a62e-9d1fccde20b3" containerName="payload-pre-processing" Nov 03 10:51:37.722643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.722062 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e1019a-4d07-44ed-935e-67d588004dd1" containerName="maas-api" Nov 03 10:51:37.722643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.722072 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="84cb267c-9d88-44ea-9178-ae6146181af1" containerName="payload-processing" Nov 03 10:51:37.722643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.722080 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ebe5575-ee5c-448e-8858-0c6e3803efe1" containerName="payload-processing" Nov 03 10:51:37.722643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.722087 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21e89d4-3943-4edb-ad24-4718c7f19ed1" containerName="payload-pre-processing" Nov 03 10:51:37.722643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.722096 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="046018fb-6293-435f-a62e-9d1fccde20b3" containerName="payload-pre-processing" Nov 03 10:51:37.722643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.722104 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="f952ba2e-5049-4a57-851c-7ee424a7b75b" containerName="maas-api" Nov 03 10:51:37.722643 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.722111 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f32e25-d4ec-4b3e-8fa0-0cd14b21b916" containerName="maas-api" Nov 03 10:51:37.727704 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.727678 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.736991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.736951 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm"] Nov 03 10:51:37.739569 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.739546 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b3dc04-openshift-default-dockercfg-4cq8t\"" Nov 03 10:51:37.842321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.842277 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.842467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.842398 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.842467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.842426 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.842467 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.842448 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.842621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.842602 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98hm\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-kube-api-access-c98hm\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.842730 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.842702 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.842850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.842735 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.842850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.842787 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.842926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.842865 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.946267 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98hm\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-kube-api-access-c98hm\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.946414 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.946455 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.946496 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.946584 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.946641 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.946744 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.946782 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.946818 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.948072 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.948718 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.948823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.948722 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.951054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.949891 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.951054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.950217 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.951621 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.951594 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.951847 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.951826 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.958137 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.958110 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:37.958333 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:37.958307 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98hm\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-kube-api-access-c98hm\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:38.071043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:38.071008 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:38.358081 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:38.358047 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm"] Nov 03 10:51:38.358508 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:51:38.358486 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cb5647_4acd_47ca_95dd_8b46b5fc8c5b.slice/crio-1de611a93c72c0a82a48866dbc17b44c6c13d0ca56c0092b8382b46eb34f71ec WatchSource:0}: Error finding container 1de611a93c72c0a82a48866dbc17b44c6c13d0ca56c0092b8382b46eb34f71ec: Status 404 returned error can't find the container with id 1de611a93c72c0a82a48866dbc17b44c6c13d0ca56c0092b8382b46eb34f71ec Nov 03 10:51:38.361154 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:38.361121 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:51:38.361247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:38.361183 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:51:38.361247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:38.361215 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:51:38.849998 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:38.849955 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" event={"ID":"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b","Type":"ContainerStarted","Data":"4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97"} Nov 03 10:51:38.850415 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:38.850071 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" event={"ID":"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b","Type":"ContainerStarted","Data":"1de611a93c72c0a82a48866dbc17b44c6c13d0ca56c0092b8382b46eb34f71ec"} Nov 03 10:51:38.905664 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:38.905609 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" podStartSLOduration=1.90559265 podStartE2EDuration="1.90559265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:38.898858193 +0000 UTC m=+1376.883729475" watchObservedRunningTime="2026-11-03 10:51:38.90559265 +0000 UTC m=+1376.890463933" Nov 03 10:51:39.071356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:39.071279 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:39.072695 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:39.072661 5086 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 10:51:39.072821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:39.072711 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 10:51:40.073288 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:40.073256 5086 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 10:51:40.073915 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:40.073886 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 10:51:41.072187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:41.072150 5086 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 10:51:41.072415 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:41.072223 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 10:51:42.071614 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:42.071547 5086 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 10:51:42.072113 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:42.071671 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 10:51:43.073264 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:43.073229 5086 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 10:51:43.073623 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:43.073333 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 10:51:43.166166 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:43.166133 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm_bc105700-2395-4afd-be20-56f83ac07a64/payload-processing/0.log" Nov 03 10:51:43.840804 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:43.840773 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh_679be1a2-3c56-467f-940c-fac577127482/payload-processing/0.log" Nov 03 10:51:44.067288 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:44.067257 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh_679be1a2-3c56-467f-940c-fac577127482/payload-processing/0.log" Nov 03 10:51:44.071437 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:44.071416 5086 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 10:51:44.071517 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:44.071490 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 10:51:45.088380 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:45.088337 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:45.090203 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:45.090172 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:45.091201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:45.091180 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:51:51.662777 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.662742 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5"] Nov 03 10:51:51.669437 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.669410 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.676646 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.676611 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-835861-serving-cert\"" Nov 03 10:51:51.694167 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.694135 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5"] Nov 03 10:51:51.707969 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.707930 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-maas-api-tls\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.708149 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.707979 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.708227 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.708148 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-metrics-certs\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.708227 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.708209 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r94mh\" (UniqueName: \"kubernetes.io/projected/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-kube-api-access-r94mh\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.813537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.810099 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-maas-api-tls\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.813537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.810141 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.813537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.810175 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-metrics-certs\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.813537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.810210 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r94mh\" (UniqueName: \"kubernetes.io/projected/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-kube-api-access-r94mh\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.813537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.811224 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.833042 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.833004 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-metrics-certs\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.835066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.835039 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-maas-api-tls\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.837593 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.837566 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r94mh\" (UniqueName: \"kubernetes.io/projected/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-kube-api-access-r94mh\") pod \"maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:51.996233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:51.996145 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:52.303575 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:52.303543 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5"] Nov 03 10:51:53.009205 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:53.009167 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" event={"ID":"d12125d2-f7f5-40c6-9f16-a2dbfab74f17","Type":"ContainerStarted","Data":"3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e"} Nov 03 10:51:53.009636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:53.009222 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" event={"ID":"d12125d2-f7f5-40c6-9f16-a2dbfab74f17","Type":"ContainerStarted","Data":"0c4d0c2a30601874ee38822fe9835a589d6468e0ca3dd133c42daec506f1edc1"} Nov 03 10:51:53.009636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:53.009252 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:51:53.028772 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:53.028705 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" podStartSLOduration=1.529650043 podStartE2EDuration="2.028686874s" totalImagesPullingTime="499.036831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:53.024775487 +0000 UTC m=+1391.009646773" watchObservedRunningTime="2026-11-03 10:51:53.028686874 +0000 UTC m=+1391.013558158" Nov 03 10:51:54.275237 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:51:54.275196 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:51:54.951079 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:54.951043 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc"] Nov 03 10:51:54.955867 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:54.955837 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:54.963615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:54.963585 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc"] Nov 03 10:51:54.966792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:54.966702 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f81a79-serving-cert\"" Nov 03 10:51:55.083316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.080700 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-metrics-certs\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.083316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.080820 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc090b06-0afb-4aed-b8cc-4b23705363ff-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.083316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.080996 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.083316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.081102 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc95m\" (UniqueName: \"kubernetes.io/projected/cc090b06-0afb-4aed-b8cc-4b23705363ff-kube-api-access-zc95m\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.182532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.182495 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.182723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.182589 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc95m\" (UniqueName: \"kubernetes.io/projected/cc090b06-0afb-4aed-b8cc-4b23705363ff-kube-api-access-zc95m\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.182723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.182651 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-metrics-certs\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.182723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.182707 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc090b06-0afb-4aed-b8cc-4b23705363ff-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.183431 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.183404 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc090b06-0afb-4aed-b8cc-4b23705363ff-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.185145 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.185119 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-metrics-certs\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.185271 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.185258 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.192411 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.192387 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc95m\" (UniqueName: \"kubernetes.io/projected/cc090b06-0afb-4aed-b8cc-4b23705363ff-kube-api-access-zc95m\") pod \"maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.282054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.281975 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:55.564075 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:55.564045 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc"] Nov 03 10:51:55.566113 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:51:55.566082 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc090b06_0afb_4aed_b8cc_4b23705363ff.slice/crio-8c73a95033c089a2f3f7ff5212fed323018ea9f8a7117a9481a02fb64d4178c5 WatchSource:0}: Error finding container 8c73a95033c089a2f3f7ff5212fed323018ea9f8a7117a9481a02fb64d4178c5: Status 404 returned error can't find the container with id 8c73a95033c089a2f3f7ff5212fed323018ea9f8a7117a9481a02fb64d4178c5 Nov 03 10:51:56.036037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:56.036008 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" event={"ID":"cc090b06-0afb-4aed-b8cc-4b23705363ff","Type":"ContainerStarted","Data":"8c73a95033c089a2f3f7ff5212fed323018ea9f8a7117a9481a02fb64d4178c5"} Nov 03 10:51:57.045590 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:57.045457 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" event={"ID":"cc090b06-0afb-4aed-b8cc-4b23705363ff","Type":"ContainerStarted","Data":"c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc"} Nov 03 10:51:57.046153 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:57.046130 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:51:57.064615 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:57.064560 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" podStartSLOduration=2.617333985 podStartE2EDuration="3.064547022s" totalImagesPullingTime="447.213037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:57.061579424 +0000 UTC m=+1395.046450708" watchObservedRunningTime="2026-11-03 10:51:57.064547022 +0000 UTC m=+1395.049418318" Nov 03 10:51:59.073489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:51:59.073457 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:52:00.938623 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:00.938588 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469"] Nov 03 10:52:00.939470 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:00.939431 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" podUID="e705d8fa-f289-456c-a228-768f165311e5" containerName="maas-api" containerID="cri-o://43db04fc76c9f4d9eca799ca538a7099a131786c6a272f7e9d775effef7d5a34" gracePeriod=30 Nov 03 10:52:01.117169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.117133 5086 generic.go:356] "Generic (PLEG): container finished" podID="e705d8fa-f289-456c-a228-768f165311e5" containerID="43db04fc76c9f4d9eca799ca538a7099a131786c6a272f7e9d775effef7d5a34" exitCode=0 Nov 03 10:52:01.117372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.117275 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" event={"ID":"e705d8fa-f289-456c-a228-768f165311e5","Type":"ContainerDied","Data":"43db04fc76c9f4d9eca799ca538a7099a131786c6a272f7e9d775effef7d5a34"} Nov 03 10:52:01.400160 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.400131 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:52:01.544112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.544076 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-metrics-certs\") pod \"e705d8fa-f289-456c-a228-768f165311e5\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " Nov 03 10:52:01.544348 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.544156 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e705d8fa-f289-456c-a228-768f165311e5-kube-api-access-2sz59\" (UniqueName: \"kubernetes.io/projected/e705d8fa-f289-456c-a228-768f165311e5-kube-api-access-2sz59\") pod \"e705d8fa-f289-456c-a228-768f165311e5\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " Nov 03 10:52:01.544348 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.544267 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e705d8fa-f289-456c-a228-768f165311e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e705d8fa-f289-456c-a228-768f165311e5-openshift-service-ca\") pod \"e705d8fa-f289-456c-a228-768f165311e5\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " Nov 03 10:52:01.544348 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.544341 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-maas-api-tls\") pod \"e705d8fa-f289-456c-a228-768f165311e5\" (UID: \"e705d8fa-f289-456c-a228-768f165311e5\") " Nov 03 10:52:01.544675 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.544651 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e705d8fa-f289-456c-a228-768f165311e5-openshift-service-ca" pod "e705d8fa-f289-456c-a228-768f165311e5" (UID: "e705d8fa-f289-456c-a228-768f165311e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:52:01.544915 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.544891 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e705d8fa-f289-456c-a228-768f165311e5-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:01.546403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.546379 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-maas-api-tls" pod "e705d8fa-f289-456c-a228-768f165311e5" (UID: "e705d8fa-f289-456c-a228-768f165311e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:52:01.546494 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.546445 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-metrics-certs" pod "e705d8fa-f289-456c-a228-768f165311e5" (UID: "e705d8fa-f289-456c-a228-768f165311e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:52:01.546494 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.546456 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e705d8fa-f289-456c-a228-768f165311e5-kube-api-access-2sz59" pod "e705d8fa-f289-456c-a228-768f165311e5" (UID: "e705d8fa-f289-456c-a228-768f165311e5"). InnerVolumeSpecName "kube-api-access-2sz59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:52:01.645897 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.645805 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:01.645897 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.645834 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e705d8fa-f289-456c-a228-768f165311e5-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:01.645897 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:01.645844 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sz59\" (UniqueName: \"kubernetes.io/projected/e705d8fa-f289-456c-a228-768f165311e5-kube-api-access-2sz59\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:02.125251 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:02.125217 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" event={"ID":"e705d8fa-f289-456c-a228-768f165311e5","Type":"ContainerDied","Data":"e2c619a55e573a93581ab65fb016eccc5ffbeee94f5ef8f286ed7c1b1ad8351b"} Nov 03 10:52:02.125251 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:02.125242 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469" Nov 03 10:52:02.125787 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:02.125288 5086 scope.go:122] "RemoveContainer" containerID="43db04fc76c9f4d9eca799ca538a7099a131786c6a272f7e9d775effef7d5a34" Nov 03 10:52:02.152396 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:02.152362 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469"] Nov 03 10:52:02.159058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:02.159029 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-814dc0-64b5fcbc79-7s469"] Nov 03 10:52:02.173490 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:02.173461 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e705d8fa-f289-456c-a228-768f165311e5" path="/var/lib/kubelet/pods/e705d8fa-f289-456c-a228-768f165311e5/volumes" Nov 03 10:52:03.059985 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:03.059960 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:52:18.413078 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.413038 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght"] Nov 03 10:52:18.413921 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.413886 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" podUID="8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" containerName="maas-api" containerID="cri-o://836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f" gracePeriod=30 Nov 03 10:52:18.478044 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.478005 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm"] Nov 03 10:52:18.478331 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.478281 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" podUID="bc105700-2395-4afd-be20-56f83ac07a64" containerName="payload-processing" containerID="cri-o://85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d" gracePeriod=30 Nov 03 10:52:18.493441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.485730 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27"] Nov 03 10:52:18.493441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.486001 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" podUID="89c04b53-cb5d-4fcf-a173-5682d67835ea" containerName="payload-pre-processing" containerID="cri-o://e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a" gracePeriod=30 Nov 03 10:52:18.887599 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.887564 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:52:18.936106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.926780 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:52:18.984372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.983599 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc105700-2395-4afd-be20-56f83ac07a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc105700-2395-4afd-be20-56f83ac07a64-plugins-config-volume\") pod \"bc105700-2395-4afd-be20-56f83ac07a64\" (UID: \"bc105700-2395-4afd-be20-56f83ac07a64\") " Nov 03 10:52:18.984372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.983733 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc105700-2395-4afd-be20-56f83ac07a64-kube-api-access-x8l9t\" (UniqueName: \"kubernetes.io/projected/bc105700-2395-4afd-be20-56f83ac07a64-kube-api-access-x8l9t\") pod \"bc105700-2395-4afd-be20-56f83ac07a64\" (UID: \"bc105700-2395-4afd-be20-56f83ac07a64\") " Nov 03 10:52:18.989318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.985412 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc105700-2395-4afd-be20-56f83ac07a64-plugins-config-volume" pod "bc105700-2395-4afd-be20-56f83ac07a64" (UID: "bc105700-2395-4afd-be20-56f83ac07a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:52:19.010435 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:18.992936 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc105700-2395-4afd-be20-56f83ac07a64-kube-api-access-x8l9t" pod "bc105700-2395-4afd-be20-56f83ac07a64" (UID: "bc105700-2395-4afd-be20-56f83ac07a64"). InnerVolumeSpecName "kube-api-access-x8l9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:52:19.075791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.075757 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:52:19.085387 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.085357 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c04b53-cb5d-4fcf-a173-5682d67835ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c04b53-cb5d-4fcf-a173-5682d67835ea-plugins-config-volume\") pod \"89c04b53-cb5d-4fcf-a173-5682d67835ea\" (UID: \"89c04b53-cb5d-4fcf-a173-5682d67835ea\") " Nov 03 10:52:19.085571 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.085457 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c04b53-cb5d-4fcf-a173-5682d67835ea-kube-api-access-l7c22\" (UniqueName: \"kubernetes.io/projected/89c04b53-cb5d-4fcf-a173-5682d67835ea-kube-api-access-l7c22\") pod \"89c04b53-cb5d-4fcf-a173-5682d67835ea\" (UID: \"89c04b53-cb5d-4fcf-a173-5682d67835ea\") " Nov 03 10:52:19.085831 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.085801 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c04b53-cb5d-4fcf-a173-5682d67835ea-plugins-config-volume" pod "89c04b53-cb5d-4fcf-a173-5682d67835ea" (UID: "89c04b53-cb5d-4fcf-a173-5682d67835ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:52:19.086484 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.086436 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc105700-2395-4afd-be20-56f83ac07a64-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:19.086484 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.086462 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8l9t\" (UniqueName: \"kubernetes.io/projected/bc105700-2395-4afd-be20-56f83ac07a64-kube-api-access-x8l9t\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:19.086484 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.086476 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c04b53-cb5d-4fcf-a173-5682d67835ea-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:19.088911 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.088366 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c04b53-cb5d-4fcf-a173-5682d67835ea-kube-api-access-l7c22" pod "89c04b53-cb5d-4fcf-a173-5682d67835ea" (UID: "89c04b53-cb5d-4fcf-a173-5682d67835ea"). InnerVolumeSpecName "kube-api-access-l7c22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:52:19.187142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.187107 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-maas-api-tls\") pod \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " Nov 03 10:52:19.187352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.187153 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-kube-api-access-qdpgr\" (UniqueName: \"kubernetes.io/projected/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-kube-api-access-qdpgr\") pod \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " Nov 03 10:52:19.187352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.187193 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-openshift-service-ca\") pod \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " Nov 03 10:52:19.187455 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.187368 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-metrics-certs\") pod \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\" (UID: \"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5\") " Nov 03 10:52:19.187652 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.187626 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-openshift-service-ca" pod "8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" (UID: "8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:52:19.188091 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.188067 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:19.188209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.188104 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7c22\" (UniqueName: \"kubernetes.io/projected/89c04b53-cb5d-4fcf-a173-5682d67835ea-kube-api-access-l7c22\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:19.189537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.189510 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-maas-api-tls" pod "8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" (UID: "8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:52:19.189537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.189523 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-kube-api-access-qdpgr" pod "8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" (UID: "8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5"). InnerVolumeSpecName "kube-api-access-qdpgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:52:19.189670 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.189541 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-metrics-certs" pod "8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" (UID: "8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:52:19.289734 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.289644 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:19.289734 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.289672 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdpgr\" (UniqueName: \"kubernetes.io/projected/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-kube-api-access-qdpgr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:19.289734 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.289688 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:19.372290 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.372253 5086 generic.go:356] "Generic (PLEG): container finished" podID="8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" containerID="836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f" exitCode=0 Nov 03 10:52:19.372518 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.372346 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" Nov 03 10:52:19.372518 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.372345 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" event={"ID":"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5","Type":"ContainerDied","Data":"836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f"} Nov 03 10:52:19.372518 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.372408 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght" event={"ID":"8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5","Type":"ContainerDied","Data":"c014d78eac9f4f414adc8a931553e741b09d6afedc3b998b8a6eceb0ce8748fd"} Nov 03 10:52:19.372518 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.372438 5086 scope.go:122] "RemoveContainer" containerID="836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f" Nov 03 10:52:19.373875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.373852 5086 generic.go:356] "Generic (PLEG): container finished" podID="bc105700-2395-4afd-be20-56f83ac07a64" containerID="85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d" exitCode=0 Nov 03 10:52:19.373986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.373901 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" Nov 03 10:52:19.373986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.373911 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" event={"ID":"bc105700-2395-4afd-be20-56f83ac07a64","Type":"ContainerDied","Data":"85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d"} Nov 03 10:52:19.373986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.373957 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm" event={"ID":"bc105700-2395-4afd-be20-56f83ac07a64","Type":"ContainerDied","Data":"bc35e2a6e8fb03fc49d11eefdefeee96aa4fba746fd8ce59481a098a2d73b595"} Nov 03 10:52:19.375421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.375397 5086 generic.go:356] "Generic (PLEG): container finished" podID="89c04b53-cb5d-4fcf-a173-5682d67835ea" containerID="e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a" exitCode=0 Nov 03 10:52:19.375563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.375434 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" Nov 03 10:52:19.375563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.375471 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" event={"ID":"89c04b53-cb5d-4fcf-a173-5682d67835ea","Type":"ContainerDied","Data":"e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a"} Nov 03 10:52:19.375563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.375504 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27" event={"ID":"89c04b53-cb5d-4fcf-a173-5682d67835ea","Type":"ContainerDied","Data":"d2ea8c0f0dccebf58747307a9aba839a61ff427b6b3e044d4457178aa41296dc"} Nov 03 10:52:19.396480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.396453 5086 scope.go:122] "RemoveContainer" containerID="836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f" Nov 03 10:52:19.396881 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:52:19.396849 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f\": container with ID starting with 836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f not found: ID does not exist" containerID="836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f" Nov 03 10:52:19.396990 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.396893 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f"} err="failed to get container status \"836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f\": rpc error: code = NotFound desc = could not find container \"836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f\": container with ID starting with 836fb6ace08b3d77b837a5e2f57f4b553860dd0af503cdaf4da3f6045b95769f not found: ID does not exist" Nov 03 10:52:19.396990 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.396911 5086 scope.go:122] "RemoveContainer" containerID="85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d" Nov 03 10:52:19.406866 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.406849 5086 scope.go:122] "RemoveContainer" containerID="85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d" Nov 03 10:52:19.407145 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:52:19.407126 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d\": container with ID starting with 85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d not found: ID does not exist" containerID="85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d" Nov 03 10:52:19.407198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.407153 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d"} err="failed to get container status \"85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d\": rpc error: code = NotFound desc = could not find container \"85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d\": container with ID starting with 85595b4abb130b9be8ccfd03ee6b50c62c2b8d4aa60420c7b936df38be68f49d not found: ID does not exist" Nov 03 10:52:19.407198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.407169 5086 scope.go:122] "RemoveContainer" containerID="e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a" Nov 03 10:52:19.416539 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.416339 5086 scope.go:122] "RemoveContainer" containerID="e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a" Nov 03 10:52:19.416794 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:52:19.416658 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a\": container with ID starting with e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a not found: ID does not exist" containerID="e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a" Nov 03 10:52:19.416794 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.416688 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a"} err="failed to get container status \"e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a\": rpc error: code = NotFound desc = could not find container \"e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a\": container with ID starting with e7650937ad7426e4709705487ab569bca5c035c9643052b13f5225db8917e03a not found: ID does not exist" Nov 03 10:52:19.423921 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.423897 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm"] Nov 03 10:52:19.432797 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.432777 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a51d55-7965b78fdd-t9jtm"] Nov 03 10:52:19.466005 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.465979 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27"] Nov 03 10:52:19.473428 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.473404 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a51d55-6458cc8bdc-xxn27"] Nov 03 10:52:19.486852 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.486823 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght"] Nov 03 10:52:19.489892 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:19.489871 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a51d55-dc57c5757-6rght"] Nov 03 10:52:20.172187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:20.172155 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c04b53-cb5d-4fcf-a173-5682d67835ea" path="/var/lib/kubelet/pods/89c04b53-cb5d-4fcf-a173-5682d67835ea/volumes" Nov 03 10:52:20.172647 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:20.172629 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" path="/var/lib/kubelet/pods/8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5/volumes" Nov 03 10:52:20.172999 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:20.172986 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc105700-2395-4afd-be20-56f83ac07a64" path="/var/lib/kubelet/pods/bc105700-2395-4afd-be20-56f83ac07a64/volumes" Nov 03 10:52:23.149926 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.149895 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj"] Nov 03 10:52:23.150584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150561 5086 cpu_manager.go:414] "containerMap: removing container" podUID="8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" containerName="maas-api" Nov 03 10:52:23.150584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150577 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" containerName="maas-api" Nov 03 10:52:23.150668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150618 5086 cpu_manager.go:414] "containerMap: removing container" podUID="89c04b53-cb5d-4fcf-a173-5682d67835ea" containerName="payload-pre-processing" Nov 03 10:52:23.150668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150624 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c04b53-cb5d-4fcf-a173-5682d67835ea" containerName="payload-pre-processing" Nov 03 10:52:23.150668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150635 5086 cpu_manager.go:414] "containerMap: removing container" podUID="e705d8fa-f289-456c-a228-768f165311e5" containerName="maas-api" Nov 03 10:52:23.150668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150640 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e705d8fa-f289-456c-a228-768f165311e5" containerName="maas-api" Nov 03 10:52:23.150668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150661 5086 cpu_manager.go:414] "containerMap: removing container" podUID="bc105700-2395-4afd-be20-56f83ac07a64" containerName="payload-processing" Nov 03 10:52:23.150668 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150666 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc105700-2395-4afd-be20-56f83ac07a64" containerName="payload-processing" Nov 03 10:52:23.150880 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150764 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c04b53-cb5d-4fcf-a173-5682d67835ea" containerName="payload-pre-processing" Nov 03 10:52:23.150880 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150773 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="e705d8fa-f289-456c-a228-768f165311e5" containerName="maas-api" Nov 03 10:52:23.150880 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150781 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc105700-2395-4afd-be20-56f83ac07a64" containerName="payload-processing" Nov 03 10:52:23.150880 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.150788 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6e6ee0-b1f0-44a8-aaaa-7e08e90c08b5" containerName="maas-api" Nov 03 10:52:23.156134 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.156108 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.167779 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.164818 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-cd6d01-serving-cert\"" Nov 03 10:52:23.177719 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.177688 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj"] Nov 03 10:52:23.244089 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.244055 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-metrics-certs\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.244272 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.244179 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9sr\" (UniqueName: \"kubernetes.io/projected/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-kube-api-access-pg9sr\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.244385 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.244367 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.244505 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.244491 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.346770 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.346715 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.346981 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.346889 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-metrics-certs\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.346981 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.346922 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9sr\" (UniqueName: \"kubernetes.io/projected/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-kube-api-access-pg9sr\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.347095 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.347046 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.347957 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.347924 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.353496 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.353467 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-metrics-certs\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.356949 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.356923 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9sr\" (UniqueName: \"kubernetes.io/projected/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-kube-api-access-pg9sr\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.357276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.357256 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.488553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.488465 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:23.800418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.797363 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj"] Nov 03 10:52:23.987341 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.987288 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8"] Nov 03 10:52:23.994255 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:23.994227 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:24.000476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.000445 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8"] Nov 03 10:52:24.068842 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.068811 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-08c392\"" Nov 03 10:52:24.069012 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.068941 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-08c392-dockercfg-6frqm\"" Nov 03 10:52:24.167421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.167380 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgkgn\" (UniqueName: \"kubernetes.io/projected/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-kube-api-access-dgkgn\") pod \"payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8\" (UID: \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:24.174634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.174586 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8\" (UID: \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:24.276741 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.276706 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgkgn\" (UniqueName: \"kubernetes.io/projected/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-kube-api-access-dgkgn\") pod \"payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8\" (UID: \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:24.276918 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.276880 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8\" (UID: \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:24.277826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.277801 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8\" (UID: \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:24.278241 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:52:24.278211 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:52:24.284612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.284580 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgkgn\" (UniqueName: \"kubernetes.io/projected/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-kube-api-access-dgkgn\") pod \"payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8\" (UID: \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:24.317169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.317130 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:24.415091 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.415050 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" event={"ID":"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0","Type":"ContainerStarted","Data":"5037d6d7132637d515d00dc9c8e1b613e8c13a80809eb447bfd6cc39898a05ca"} Nov 03 10:52:24.647609 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:24.647582 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8"] Nov 03 10:52:24.647958 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:52:24.647928 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a64a74f_43e0_4695_8fb5_f63cbabed4e6.slice/crio-01567c605c2e33ed98feb8ecf1d4897e6d310bb8d5c49c7358bb514d707bb25e WatchSource:0}: Error finding container 01567c605c2e33ed98feb8ecf1d4897e6d310bb8d5c49c7358bb514d707bb25e: Status 404 returned error can't find the container with id 01567c605c2e33ed98feb8ecf1d4897e6d310bb8d5c49c7358bb514d707bb25e Nov 03 10:52:25.423489 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:25.423449 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" event={"ID":"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0","Type":"ContainerStarted","Data":"3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8"} Nov 03 10:52:25.423951 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:25.423521 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:25.425069 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:25.425039 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" event={"ID":"4a64a74f-43e0-4695-8fb5-f63cbabed4e6","Type":"ContainerStarted","Data":"e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9"} Nov 03 10:52:25.425202 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:25.425088 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" event={"ID":"4a64a74f-43e0-4695-8fb5-f63cbabed4e6","Type":"ContainerStarted","Data":"01567c605c2e33ed98feb8ecf1d4897e6d310bb8d5c49c7358bb514d707bb25e"} Nov 03 10:52:25.425202 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:25.425117 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:25.445127 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:25.445079 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" podStartSLOduration=1.890041885 podStartE2EDuration="2.445066737s" totalImagesPullingTime="555.024852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:25.442234775 +0000 UTC m=+1423.427106060" watchObservedRunningTime="2026-11-03 10:52:25.445066737 +0000 UTC m=+1423.429938019" Nov 03 10:52:25.460809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:25.460763 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" podStartSLOduration=2.460749693 podStartE2EDuration="2.460749693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:25.458613391 +0000 UTC m=+1423.443484674" watchObservedRunningTime="2026-11-03 10:52:25.460749693 +0000 UTC m=+1423.445620976" Nov 03 10:52:30.656135 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.656089 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw"] Nov 03 10:52:30.667106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.666330 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw"] Nov 03 10:52:30.667106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.666452 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.675257 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.675231 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-08c392-serving-cert\"" Nov 03 10:52:30.677352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.677275 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.677458 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.677352 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgt65\" (UniqueName: \"kubernetes.io/projected/a91ce149-c020-4fb1-bd6e-384ef224bb2a-kube-api-access-sgt65\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.677578 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.677555 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-metrics-certs\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.677717 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.677696 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a91ce149-c020-4fb1-bd6e-384ef224bb2a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.784198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.782680 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a91ce149-c020-4fb1-bd6e-384ef224bb2a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.784198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.782858 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.784198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.782907 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgt65\" (UniqueName: \"kubernetes.io/projected/a91ce149-c020-4fb1-bd6e-384ef224bb2a-kube-api-access-sgt65\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.784198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.783040 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-metrics-certs\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.785597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.785569 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a91ce149-c020-4fb1-bd6e-384ef224bb2a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.791573 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.791537 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-metrics-certs\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.792662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.792223 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.793602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.793579 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgt65\" (UniqueName: \"kubernetes.io/projected/a91ce149-c020-4fb1-bd6e-384ef224bb2a-kube-api-access-sgt65\") pod \"maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:30.986193 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:30.986105 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:31.269641 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:31.269602 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw"] Nov 03 10:52:31.445728 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:31.445696 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:52:31.485255 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:31.485204 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" event={"ID":"a91ce149-c020-4fb1-bd6e-384ef224bb2a","Type":"ContainerStarted","Data":"0fa404f2b366faea5701ccaa866c980f355ccb7683c82d07a4cddc7a5c9a0ff9"} Nov 03 10:52:32.496398 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:32.496362 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" event={"ID":"a91ce149-c020-4fb1-bd6e-384ef224bb2a","Type":"ContainerStarted","Data":"4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287"} Nov 03 10:52:32.496815 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:32.496486 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:32.513270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:32.513216 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" podStartSLOduration=2.137623885 podStartE2EDuration="2.513201462s" totalImagesPullingTime="375.577577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:32.512336103 +0000 UTC m=+1430.497207386" watchObservedRunningTime="2026-11-03 10:52:32.513201462 +0000 UTC m=+1430.498072767" Nov 03 10:52:33.805639 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:33.805516 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s"] Nov 03 10:52:33.806107 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:33.805788 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" podUID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerName="maas-api" containerID="cri-o://0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02" gracePeriod=30 Nov 03 10:52:34.119895 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.119870 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:52:34.151814 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.151781 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-maas-api-tls\") pod \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " Nov 03 10:52:34.152000 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.151957 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a09f09-e2e9-4356-b3fb-5860237c80cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a09f09-e2e9-4356-b3fb-5860237c80cf-openshift-service-ca\") pod \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " Nov 03 10:52:34.152131 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.152112 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a09f09-e2e9-4356-b3fb-5860237c80cf-kube-api-access-86lt7\" (UniqueName: \"kubernetes.io/projected/d8a09f09-e2e9-4356-b3fb-5860237c80cf-kube-api-access-86lt7\") pod \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " Nov 03 10:52:34.152215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.152167 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-metrics-certs\") pod \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\" (UID: \"d8a09f09-e2e9-4356-b3fb-5860237c80cf\") " Nov 03 10:52:34.152893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.152408 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a09f09-e2e9-4356-b3fb-5860237c80cf-openshift-service-ca" pod "d8a09f09-e2e9-4356-b3fb-5860237c80cf" (UID: "d8a09f09-e2e9-4356-b3fb-5860237c80cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:52:34.152893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.152855 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a09f09-e2e9-4356-b3fb-5860237c80cf-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:34.154565 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.154531 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a09f09-e2e9-4356-b3fb-5860237c80cf-kube-api-access-86lt7" pod "d8a09f09-e2e9-4356-b3fb-5860237c80cf" (UID: "d8a09f09-e2e9-4356-b3fb-5860237c80cf"). InnerVolumeSpecName "kube-api-access-86lt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:52:34.154680 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.154644 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-maas-api-tls" pod "d8a09f09-e2e9-4356-b3fb-5860237c80cf" (UID: "d8a09f09-e2e9-4356-b3fb-5860237c80cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:52:34.154743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.154718 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-metrics-certs" pod "d8a09f09-e2e9-4356-b3fb-5860237c80cf" (UID: "d8a09f09-e2e9-4356-b3fb-5860237c80cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:52:34.267033 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.266951 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86lt7\" (UniqueName: \"kubernetes.io/projected/d8a09f09-e2e9-4356-b3fb-5860237c80cf-kube-api-access-86lt7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:34.267033 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.266991 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:34.267033 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.267005 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a09f09-e2e9-4356-b3fb-5860237c80cf-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:34.513534 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.513452 5086 generic.go:356] "Generic (PLEG): container finished" podID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerID="0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02" exitCode=0 Nov 03 10:52:34.513534 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.513517 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" Nov 03 10:52:34.513716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.513529 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" event={"ID":"d8a09f09-e2e9-4356-b3fb-5860237c80cf","Type":"ContainerDied","Data":"0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02"} Nov 03 10:52:34.513716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.513584 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s" event={"ID":"d8a09f09-e2e9-4356-b3fb-5860237c80cf","Type":"ContainerDied","Data":"9dba72543a00c2329248d8fd05f1a2b57e1f3d0de7e6660fc52fb421b4f40fb0"} Nov 03 10:52:34.513716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.513605 5086 scope.go:122] "RemoveContainer" containerID="0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02" Nov 03 10:52:34.525165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.525140 5086 scope.go:122] "RemoveContainer" containerID="46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554" Nov 03 10:52:34.534771 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.534741 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s"] Nov 03 10:52:34.535385 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.535362 5086 scope.go:122] "RemoveContainer" containerID="0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02" Nov 03 10:52:34.535696 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:52:34.535670 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02\": container with ID starting with 0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02 not found: ID does not exist" containerID="0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02" Nov 03 10:52:34.535768 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.535712 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02"} err="failed to get container status \"0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02\": rpc error: code = NotFound desc = could not find container \"0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02\": container with ID starting with 0234209076d631756bddd26fa0e0e0c9562a9b9aaaef8520c132cbc4ce16bd02 not found: ID does not exist" Nov 03 10:52:34.535768 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.535736 5086 scope.go:122] "RemoveContainer" containerID="46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554" Nov 03 10:52:34.536040 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:52:34.536017 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554\": container with ID starting with 46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554 not found: ID does not exist" containerID="46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554" Nov 03 10:52:34.536129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.536049 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554"} err="failed to get container status \"46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554\": rpc error: code = NotFound desc = could not find container \"46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554\": container with ID starting with 46ffe3d4ab703f15379d4e3211594cf78ffa31bb27970ee11a1eb73b8db40554 not found: ID does not exist" Nov 03 10:52:34.538586 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:34.538566 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3dc04-66b9c848c6-f7h8s"] Nov 03 10:52:36.172133 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:36.172089 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" path="/var/lib/kubelet/pods/d8a09f09-e2e9-4356-b3fb-5860237c80cf/volumes" Nov 03 10:52:36.437160 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:36.437073 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:52:38.510964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:38.510934 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:52:41.122037 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:41.122003 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm"] Nov 03 10:52:41.122552 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:41.122251 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" containerID="cri-o://4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97" gracePeriod=30 Nov 03 10:52:45.093906 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:45.093853 5086 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 10:52:45.094514 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:45.093933 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 10:52:46.414542 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.414515 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:52:46.516154 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516120 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-credential-socket\") pod \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " Nov 03 10:52:46.516345 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516192 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-certs\") pod \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " Nov 03 10:52:46.516345 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516272 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-kube-api-access-c98hm\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-kube-api-access-c98hm\") pod \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " Nov 03 10:52:46.516424 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516353 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-token\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-token\") pod \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " Nov 03 10:52:46.516424 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516392 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-podinfo\") pod \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " Nov 03 10:52:46.516493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516427 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-credential-socket" pod "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" (UID: "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:52:46.516543 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516489 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-envoy\") pod \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " Nov 03 10:52:46.516576 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516548 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istiod-ca-cert\") pod \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " Nov 03 10:52:46.516576 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516553 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-certs" pod "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" (UID: "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:52:46.516685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516667 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-socket\") pod \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " Nov 03 10:52:46.516752 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.516736 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-data\") pod \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\" (UID: \"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b\") " Nov 03 10:52:46.517474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.517242 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-socket" pod "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" (UID: "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:52:46.517474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.517426 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-data" pod "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" (UID: "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:52:46.517685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.517578 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:46.517685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.517597 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:46.517685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.517610 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:46.517685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.517623 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:46.517937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.517917 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istiod-ca-cert" pod "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" (UID: "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:52:46.518818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.518790 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-envoy" pod "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" (UID: "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:52:46.518818 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.518809 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-podinfo" pod "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" (UID: "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:52:46.519084 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.519066 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-token" pod "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" (UID: "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:52:46.519153 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.519093 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-kube-api-access-c98hm" pod "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" (UID: "96cb5647-4acd-47ca-95dd-8b46b5fc8c5b"). InnerVolumeSpecName "kube-api-access-c98hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:52:46.619267 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619220 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98hm\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-kube-api-access-c98hm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:46.619267 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619260 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:46.619267 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619275 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:46.619597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619290 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:46.619597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619363 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:52:46.619597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619481 5086 generic.go:356] "Generic (PLEG): container finished" podID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerID="4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97" exitCode=0 Nov 03 10:52:46.619597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619545 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" Nov 03 10:52:46.619597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619562 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" event={"ID":"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b","Type":"ContainerDied","Data":"4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97"} Nov 03 10:52:46.619892 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619633 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm" event={"ID":"96cb5647-4acd-47ca-95dd-8b46b5fc8c5b","Type":"ContainerDied","Data":"1de611a93c72c0a82a48866dbc17b44c6c13d0ca56c0092b8382b46eb34f71ec"} Nov 03 10:52:46.619892 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.619681 5086 scope.go:122] "RemoveContainer" containerID="4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97" Nov 03 10:52:46.632953 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.632936 5086 scope.go:122] "RemoveContainer" containerID="4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97" Nov 03 10:52:46.633243 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:52:46.633225 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97\": container with ID starting with 4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97 not found: ID does not exist" containerID="4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97" Nov 03 10:52:46.633352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.633252 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97"} err="failed to get container status \"4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97\": rpc error: code = NotFound desc = could not find container \"4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97\": container with ID starting with 4dcd5d55f087d959f237c4f50e265666bbf5176e529635ee5ac0090b2fd83f97 not found: ID does not exist" Nov 03 10:52:46.660199 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.660055 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm"] Nov 03 10:52:46.664189 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:46.664154 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-vh2rm"] Nov 03 10:52:48.172894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:48.172862 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" path="/var/lib/kubelet/pods/96cb5647-4acd-47ca-95dd-8b46b5fc8c5b/volumes" Nov 03 10:52:49.456858 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.456820 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms"] Nov 03 10:52:49.457929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.457905 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerName="maas-api" Nov 03 10:52:49.457929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.457929 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerName="maas-api" Nov 03 10:52:49.458109 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.458001 5086 cpu_manager.go:414] "containerMap: removing container" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" Nov 03 10:52:49.458109 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.458010 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" Nov 03 10:52:49.458109 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.458042 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerName="maas-api" Nov 03 10:52:49.458109 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.458051 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerName="maas-api" Nov 03 10:52:49.458367 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.458200 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cb5647-4acd-47ca-95dd-8b46b5fc8c5b" containerName="istio-proxy" Nov 03 10:52:49.458367 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.458215 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerName="maas-api" Nov 03 10:52:49.459171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.459149 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a09f09-e2e9-4356-b3fb-5860237c80cf" containerName="maas-api" Nov 03 10:52:49.477997 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.465453 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.477997 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.470216 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms"] Nov 03 10:52:49.477997 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.477148 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a35a97-serving-cert\"" Nov 03 10:52:49.575483 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.575448 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cq5k\" (UniqueName: \"kubernetes.io/projected/5304f031-410d-4251-8c2b-2391978c4884-kube-api-access-2cq5k\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.575683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.575516 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-metrics-certs\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.575756 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.575686 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.575864 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.575846 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5304f031-410d-4251-8c2b-2391978c4884-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.677273 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.677242 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-metrics-certs\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.677506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.677356 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.677506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.677458 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5304f031-410d-4251-8c2b-2391978c4884-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.677633 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.677554 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cq5k\" (UniqueName: \"kubernetes.io/projected/5304f031-410d-4251-8c2b-2391978c4884-kube-api-access-2cq5k\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.678215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.678190 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5304f031-410d-4251-8c2b-2391978c4884-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.680058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.680034 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-metrics-certs\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.680325 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.680267 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.686176 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.686145 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cq5k\" (UniqueName: \"kubernetes.io/projected/5304f031-410d-4251-8c2b-2391978c4884-kube-api-access-2cq5k\") pod \"maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:49.797609 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:49.797518 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:50.078970 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:52:50.078934 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5304f031_410d_4251_8c2b_2391978c4884.slice/crio-027d927763d4deb1014b25da8143f47cd9aa981e0255a6928c0517b45786de58 WatchSource:0}: Error finding container 027d927763d4deb1014b25da8143f47cd9aa981e0255a6928c0517b45786de58: Status 404 returned error can't find the container with id 027d927763d4deb1014b25da8143f47cd9aa981e0255a6928c0517b45786de58 Nov 03 10:52:50.079187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:50.079164 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms"] Nov 03 10:52:50.684146 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:50.684105 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" event={"ID":"5304f031-410d-4251-8c2b-2391978c4884","Type":"ContainerStarted","Data":"447642f448751a4081c82acf82062a27cb045082a142114633369b392d1271d2"} Nov 03 10:52:50.684685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:50.684181 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" event={"ID":"5304f031-410d-4251-8c2b-2391978c4884","Type":"ContainerStarted","Data":"027d927763d4deb1014b25da8143f47cd9aa981e0255a6928c0517b45786de58"} Nov 03 10:52:50.684685 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:50.684221 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:52:50.705626 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:50.705571 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" podStartSLOduration=1.209857642 podStartE2EDuration="1.705557381s" totalImagesPullingTime="495.699739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:50.7007075 +0000 UTC m=+1448.685578784" watchObservedRunningTime="2026-11-03 10:52:50.705557381 +0000 UTC m=+1448.690428664" Nov 03 10:52:54.275243 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:52:54.275202 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:52:56.694229 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:52:56.694188 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" podUID="5304f031-410d-4251-8c2b-2391978c4884" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.146:8443/health\": dial tcp 10.132.0.146:8443: connect: connection refused" Nov 03 10:53:00.791730 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:00.791685 5086 generic.go:356] "Generic (PLEG): container finished" podID="5304f031-410d-4251-8c2b-2391978c4884" containerID="447642f448751a4081c82acf82062a27cb045082a142114633369b392d1271d2" exitCode=1 Nov 03 10:53:00.792141 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:00.791749 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" event={"ID":"5304f031-410d-4251-8c2b-2391978c4884","Type":"ContainerDied","Data":"447642f448751a4081c82acf82062a27cb045082a142114633369b392d1271d2"} Nov 03 10:53:00.792406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:00.792384 5086 scope.go:122] "RemoveContainer" containerID="447642f448751a4081c82acf82062a27cb045082a142114633369b392d1271d2" Nov 03 10:53:01.801194 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:01.801158 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" event={"ID":"5304f031-410d-4251-8c2b-2391978c4884","Type":"ContainerStarted","Data":"dd0762fda1f194244c854e008a5283e70c1092c06c72f0f33275e0fab49f8618"} Nov 03 10:53:01.801660 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:01.801361 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:53:03.197075 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.197038 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5"] Nov 03 10:53:03.197665 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.197290 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" podUID="d12125d2-f7f5-40c6-9f16-a2dbfab74f17" containerName="maas-api" containerID="cri-o://3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e" gracePeriod=30 Nov 03 10:53:03.587142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.587110 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:53:03.730689 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.730656 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-openshift-service-ca\") pod \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " Nov 03 10:53:03.730934 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.730770 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-kube-api-access-r94mh\" (UniqueName: \"kubernetes.io/projected/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-kube-api-access-r94mh\") pod \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " Nov 03 10:53:03.730934 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.730816 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-metrics-certs\") pod \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " Nov 03 10:53:03.731032 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.730942 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-maas-api-tls\") pod \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\" (UID: \"d12125d2-f7f5-40c6-9f16-a2dbfab74f17\") " Nov 03 10:53:03.731077 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.731050 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-openshift-service-ca" pod "d12125d2-f7f5-40c6-9f16-a2dbfab74f17" (UID: "d12125d2-f7f5-40c6-9f16-a2dbfab74f17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:53:03.731538 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.731515 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:03.733191 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.733164 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-maas-api-tls" pod "d12125d2-f7f5-40c6-9f16-a2dbfab74f17" (UID: "d12125d2-f7f5-40c6-9f16-a2dbfab74f17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:53:03.733586 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.733546 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-metrics-certs" pod "d12125d2-f7f5-40c6-9f16-a2dbfab74f17" (UID: "d12125d2-f7f5-40c6-9f16-a2dbfab74f17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:53:03.733690 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.733590 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-kube-api-access-r94mh" pod "d12125d2-f7f5-40c6-9f16-a2dbfab74f17" (UID: "d12125d2-f7f5-40c6-9f16-a2dbfab74f17"). InnerVolumeSpecName "kube-api-access-r94mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:53:03.816153 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.816113 5086 generic.go:356] "Generic (PLEG): container finished" podID="d12125d2-f7f5-40c6-9f16-a2dbfab74f17" containerID="3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e" exitCode=0 Nov 03 10:53:03.816390 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.816174 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" Nov 03 10:53:03.816390 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.816192 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" event={"ID":"d12125d2-f7f5-40c6-9f16-a2dbfab74f17","Type":"ContainerDied","Data":"3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e"} Nov 03 10:53:03.816390 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.816266 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5" event={"ID":"d12125d2-f7f5-40c6-9f16-a2dbfab74f17","Type":"ContainerDied","Data":"0c4d0c2a30601874ee38822fe9835a589d6468e0ca3dd133c42daec506f1edc1"} Nov 03 10:53:03.816390 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.816311 5086 scope.go:122] "RemoveContainer" containerID="3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e" Nov 03 10:53:03.828889 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.828868 5086 scope.go:122] "RemoveContainer" containerID="3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e" Nov 03 10:53:03.829482 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:03.829455 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e\": container with ID starting with 3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e not found: ID does not exist" containerID="3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e" Nov 03 10:53:03.829582 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.829493 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e"} err="failed to get container status \"3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e\": rpc error: code = NotFound desc = could not find container \"3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e\": container with ID starting with 3962ab3782c2e277845e4b881226ce60dc2f6f62aaa4f8cda7c9b2e225d17a8e not found: ID does not exist" Nov 03 10:53:03.833356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.833335 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r94mh\" (UniqueName: \"kubernetes.io/projected/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-kube-api-access-r94mh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:03.833446 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.833358 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:03.833446 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.833368 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12125d2-f7f5-40c6-9f16-a2dbfab74f17-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:03.841059 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.841035 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5"] Nov 03 10:53:03.844337 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:03.844311 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-835861-56bc9f7755-mc6v5"] Nov 03 10:53:04.173522 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:04.173448 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12125d2-f7f5-40c6-9f16-a2dbfab74f17" path="/var/lib/kubelet/pods/d12125d2-f7f5-40c6-9f16-a2dbfab74f17/volumes" Nov 03 10:53:05.309592 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.309552 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc"] Nov 03 10:53:05.310461 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.309847 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" podUID="cc090b06-0afb-4aed-b8cc-4b23705363ff" containerName="maas-api" containerID="cri-o://c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc" gracePeriod=30 Nov 03 10:53:05.676327 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.675338 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:53:05.770742 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.770709 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-metrics-certs\") pod \"cc090b06-0afb-4aed-b8cc-4b23705363ff\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " Nov 03 10:53:05.770905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.770834 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc090b06-0afb-4aed-b8cc-4b23705363ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc090b06-0afb-4aed-b8cc-4b23705363ff-openshift-service-ca\") pod \"cc090b06-0afb-4aed-b8cc-4b23705363ff\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " Nov 03 10:53:05.771005 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.770907 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc090b06-0afb-4aed-b8cc-4b23705363ff-kube-api-access-zc95m\" (UniqueName: \"kubernetes.io/projected/cc090b06-0afb-4aed-b8cc-4b23705363ff-kube-api-access-zc95m\") pod \"cc090b06-0afb-4aed-b8cc-4b23705363ff\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " Nov 03 10:53:05.771068 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.771057 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-maas-api-tls\") pod \"cc090b06-0afb-4aed-b8cc-4b23705363ff\" (UID: \"cc090b06-0afb-4aed-b8cc-4b23705363ff\") " Nov 03 10:53:05.771363 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.771270 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc090b06-0afb-4aed-b8cc-4b23705363ff-openshift-service-ca" pod "cc090b06-0afb-4aed-b8cc-4b23705363ff" (UID: "cc090b06-0afb-4aed-b8cc-4b23705363ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:53:05.771990 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.771942 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc090b06-0afb-4aed-b8cc-4b23705363ff-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:05.775502 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.775458 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-metrics-certs" pod "cc090b06-0afb-4aed-b8cc-4b23705363ff" (UID: "cc090b06-0afb-4aed-b8cc-4b23705363ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:53:05.775630 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.775543 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-maas-api-tls" pod "cc090b06-0afb-4aed-b8cc-4b23705363ff" (UID: "cc090b06-0afb-4aed-b8cc-4b23705363ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:53:05.778059 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.778026 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc090b06-0afb-4aed-b8cc-4b23705363ff-kube-api-access-zc95m" pod "cc090b06-0afb-4aed-b8cc-4b23705363ff" (UID: "cc090b06-0afb-4aed-b8cc-4b23705363ff"). InnerVolumeSpecName "kube-api-access-zc95m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:53:05.834732 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.834621 5086 generic.go:356] "Generic (PLEG): container finished" podID="cc090b06-0afb-4aed-b8cc-4b23705363ff" containerID="c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc" exitCode=0 Nov 03 10:53:05.834732 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.834718 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" event={"ID":"cc090b06-0afb-4aed-b8cc-4b23705363ff","Type":"ContainerDied","Data":"c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc"} Nov 03 10:53:05.835003 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.834773 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" event={"ID":"cc090b06-0afb-4aed-b8cc-4b23705363ff","Type":"ContainerDied","Data":"8c73a95033c089a2f3f7ff5212fed323018ea9f8a7117a9481a02fb64d4178c5"} Nov 03 10:53:05.835003 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.834804 5086 scope.go:122] "RemoveContainer" containerID="c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc" Nov 03 10:53:05.835003 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.834718 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc" Nov 03 10:53:05.848857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.848827 5086 scope.go:122] "RemoveContainer" containerID="c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc" Nov 03 10:53:05.849153 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:05.849126 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc\": container with ID starting with c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc not found: ID does not exist" containerID="c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc" Nov 03 10:53:05.849249 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.849163 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc"} err="failed to get container status \"c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc\": rpc error: code = NotFound desc = could not find container \"c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc\": container with ID starting with c264203922105e129cf0dd9cd63b74cc4c1aa73031a4c092606eedd1975461bc not found: ID does not exist" Nov 03 10:53:05.870461 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.870435 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc"] Nov 03 10:53:05.873490 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.873467 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:05.873585 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.873490 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc090b06-0afb-4aed-b8cc-4b23705363ff-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:05.873585 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.873507 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc95m\" (UniqueName: \"kubernetes.io/projected/cc090b06-0afb-4aed-b8cc-4b23705363ff-kube-api-access-zc95m\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:05.897618 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:05.897586 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f81a79-7876dcbc65-2lmxc"] Nov 03 10:53:06.172241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:06.172154 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc090b06-0afb-4aed-b8cc-4b23705363ff" path="/var/lib/kubelet/pods/cc090b06-0afb-4aed-b8cc-4b23705363ff/volumes" Nov 03 10:53:07.813980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:07.813951 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:53:21.131630 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:21.131595 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8"] Nov 03 10:53:23.364271 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.364185 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw"] Nov 03 10:53:23.364810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.364458 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" podUID="a91ce149-c020-4fb1-bd6e-384ef224bb2a" containerName="maas-api" containerID="cri-o://4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287" gracePeriod=30 Nov 03 10:53:23.505228 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.505182 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" podUID="a91ce149-c020-4fb1-bd6e-384ef224bb2a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.145:8443/health\": dial tcp 10.132.0.145:8443: connect: connection refused" Nov 03 10:53:23.768371 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.768337 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:53:23.894745 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.894710 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91ce149-c020-4fb1-bd6e-384ef224bb2a-kube-api-access-sgt65\" (UniqueName: \"kubernetes.io/projected/a91ce149-c020-4fb1-bd6e-384ef224bb2a-kube-api-access-sgt65\") pod \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " Nov 03 10:53:23.894899 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.894825 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-maas-api-tls\") pod \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " Nov 03 10:53:23.894899 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.894867 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91ce149-c020-4fb1-bd6e-384ef224bb2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a91ce149-c020-4fb1-bd6e-384ef224bb2a-openshift-service-ca\") pod \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " Nov 03 10:53:23.895005 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.894989 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-metrics-certs\") pod \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\" (UID: \"a91ce149-c020-4fb1-bd6e-384ef224bb2a\") " Nov 03 10:53:23.895379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.895311 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91ce149-c020-4fb1-bd6e-384ef224bb2a-openshift-service-ca" pod "a91ce149-c020-4fb1-bd6e-384ef224bb2a" (UID: "a91ce149-c020-4fb1-bd6e-384ef224bb2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:53:23.895608 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.895590 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a91ce149-c020-4fb1-bd6e-384ef224bb2a-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:23.896937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.896911 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91ce149-c020-4fb1-bd6e-384ef224bb2a-kube-api-access-sgt65" pod "a91ce149-c020-4fb1-bd6e-384ef224bb2a" (UID: "a91ce149-c020-4fb1-bd6e-384ef224bb2a"). InnerVolumeSpecName "kube-api-access-sgt65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:53:23.897061 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.896939 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-maas-api-tls" pod "a91ce149-c020-4fb1-bd6e-384ef224bb2a" (UID: "a91ce149-c020-4fb1-bd6e-384ef224bb2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:53:23.897061 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.896959 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-metrics-certs" pod "a91ce149-c020-4fb1-bd6e-384ef224bb2a" (UID: "a91ce149-c020-4fb1-bd6e-384ef224bb2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:53:23.986644 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.986553 5086 generic.go:356] "Generic (PLEG): container finished" podID="a91ce149-c020-4fb1-bd6e-384ef224bb2a" containerID="4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287" exitCode=0 Nov 03 10:53:23.986644 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.986633 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" Nov 03 10:53:23.986895 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.986699 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" event={"ID":"a91ce149-c020-4fb1-bd6e-384ef224bb2a","Type":"ContainerDied","Data":"4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287"} Nov 03 10:53:23.986895 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.986747 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw" event={"ID":"a91ce149-c020-4fb1-bd6e-384ef224bb2a","Type":"ContainerDied","Data":"0fa404f2b366faea5701ccaa866c980f355ccb7683c82d07a4cddc7a5c9a0ff9"} Nov 03 10:53:23.986895 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.986776 5086 scope.go:122] "RemoveContainer" containerID="4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287" Nov 03 10:53:23.997697 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.997660 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:23.997939 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.997908 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgt65\" (UniqueName: \"kubernetes.io/projected/a91ce149-c020-4fb1-bd6e-384ef224bb2a-kube-api-access-sgt65\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:23.997939 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:23.997937 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a91ce149-c020-4fb1-bd6e-384ef224bb2a-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:24.002791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:24.002765 5086 scope.go:122] "RemoveContainer" containerID="4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287" Nov 03 10:53:24.004859 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:24.004833 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287\": container with ID starting with 4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287 not found: ID does not exist" containerID="4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287" Nov 03 10:53:24.005001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:24.004873 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287"} err="failed to get container status \"4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287\": rpc error: code = NotFound desc = could not find container \"4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287\": container with ID starting with 4a0b133284accdf9246d9e4dd21732e2362336c966bbce0c303044c594e88287 not found: ID does not exist" Nov 03 10:53:24.019350 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:24.019317 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw"] Nov 03 10:53:24.022616 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:24.022594 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-08c392-65c697cf96-8jcrw"] Nov 03 10:53:24.174083 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:24.174044 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91ce149-c020-4fb1-bd6e-384ef224bb2a" path="/var/lib/kubelet/pods/a91ce149-c020-4fb1-bd6e-384ef224bb2a/volumes" Nov 03 10:53:24.274819 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:24.274726 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:53:26.132357 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:26.132283 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" podUID="4a64a74f-43e0-4695-8fb5-f63cbabed4e6" containerName="payload-processing" containerID="cri-o://e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9" gracePeriod=25 Nov 03 10:53:26.414605 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:26.414577 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:53:26.540490 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:26.540460 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-kube-api-access-dgkgn\" (UniqueName: \"kubernetes.io/projected/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-kube-api-access-dgkgn\") pod \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\" (UID: \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\") " Nov 03 10:53:26.540667 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:26.540507 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-plugins-config-volume\") pod \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\" (UID: \"4a64a74f-43e0-4695-8fb5-f63cbabed4e6\") " Nov 03 10:53:26.541002 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:26.540960 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-plugins-config-volume" pod "4a64a74f-43e0-4695-8fb5-f63cbabed4e6" (UID: "4a64a74f-43e0-4695-8fb5-f63cbabed4e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:53:26.541224 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:26.541213 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:26.542778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:26.542756 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-kube-api-access-dgkgn" pod "4a64a74f-43e0-4695-8fb5-f63cbabed4e6" (UID: "4a64a74f-43e0-4695-8fb5-f63cbabed4e6"). InnerVolumeSpecName "kube-api-access-dgkgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:53:26.646189 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:26.646040 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgkgn\" (UniqueName: \"kubernetes.io/projected/4a64a74f-43e0-4695-8fb5-f63cbabed4e6-kube-api-access-dgkgn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:27.010474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:27.010385 5086 generic.go:356] "Generic (PLEG): container finished" podID="4a64a74f-43e0-4695-8fb5-f63cbabed4e6" containerID="e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9" exitCode=0 Nov 03 10:53:27.010474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:27.010441 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" Nov 03 10:53:27.010474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:27.010457 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" event={"ID":"4a64a74f-43e0-4695-8fb5-f63cbabed4e6","Type":"ContainerDied","Data":"e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9"} Nov 03 10:53:27.010792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:27.010506 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8" event={"ID":"4a64a74f-43e0-4695-8fb5-f63cbabed4e6","Type":"ContainerDied","Data":"01567c605c2e33ed98feb8ecf1d4897e6d310bb8d5c49c7358bb514d707bb25e"} Nov 03 10:53:27.010792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:27.010530 5086 scope.go:122] "RemoveContainer" containerID="e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9" Nov 03 10:53:27.028511 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:27.026978 5086 scope.go:122] "RemoveContainer" containerID="e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9" Nov 03 10:53:27.030054 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:27.029975 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9\": container with ID starting with e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9 not found: ID does not exist" containerID="e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9" Nov 03 10:53:27.030054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:27.030021 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9"} err="failed to get container status \"e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9\": rpc error: code = NotFound desc = could not find container \"e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9\": container with ID starting with e5abb58f50ec8338caeb4143c58f88811513af8a6593987543aada66fcdb64d9 not found: ID does not exist" Nov 03 10:53:27.040684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:27.040635 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8"] Nov 03 10:53:27.044597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:27.044567 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-08c392-79b8b7f779-zcqh8"] Nov 03 10:53:28.172068 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.172033 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a64a74f-43e0-4695-8fb5-f63cbabed4e6" path="/var/lib/kubelet/pods/4a64a74f-43e0-4695-8fb5-f63cbabed4e6/volumes" Nov 03 10:53:28.956505 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.956474 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn"] Nov 03 10:53:28.957119 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957107 5086 cpu_manager.go:414] "containerMap: removing container" podUID="a91ce149-c020-4fb1-bd6e-384ef224bb2a" containerName="maas-api" Nov 03 10:53:28.957172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957120 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91ce149-c020-4fb1-bd6e-384ef224bb2a" containerName="maas-api" Nov 03 10:53:28.957172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957143 5086 cpu_manager.go:414] "containerMap: removing container" podUID="cc090b06-0afb-4aed-b8cc-4b23705363ff" containerName="maas-api" Nov 03 10:53:28.957172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957148 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc090b06-0afb-4aed-b8cc-4b23705363ff" containerName="maas-api" Nov 03 10:53:28.957172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957157 5086 cpu_manager.go:414] "containerMap: removing container" podUID="4a64a74f-43e0-4695-8fb5-f63cbabed4e6" containerName="payload-processing" Nov 03 10:53:28.957172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957162 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a64a74f-43e0-4695-8fb5-f63cbabed4e6" containerName="payload-processing" Nov 03 10:53:28.957172 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957172 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d12125d2-f7f5-40c6-9f16-a2dbfab74f17" containerName="maas-api" Nov 03 10:53:28.957380 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957177 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12125d2-f7f5-40c6-9f16-a2dbfab74f17" containerName="maas-api" Nov 03 10:53:28.957380 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957267 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91ce149-c020-4fb1-bd6e-384ef224bb2a" containerName="maas-api" Nov 03 10:53:28.957380 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957279 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc090b06-0afb-4aed-b8cc-4b23705363ff" containerName="maas-api" Nov 03 10:53:28.957380 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957288 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a64a74f-43e0-4695-8fb5-f63cbabed4e6" containerName="payload-processing" Nov 03 10:53:28.957380 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.957310 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12125d2-f7f5-40c6-9f16-a2dbfab74f17" containerName="maas-api" Nov 03 10:53:28.962106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.962084 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:28.967589 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.967567 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn"] Nov 03 10:53:28.968838 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.968819 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-cd6d01\"/\"default-dockercfg-d8btb\"" Nov 03 10:53:28.969939 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.969916 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-cd6d01\"/\"praxis-route-cd6d01-kserve-self-signed-certs\"" Nov 03 10:53:28.975208 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.975189 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-cd6d01\"/\"kube-root-ca.crt\"" Nov 03 10:53:28.976862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:28.976837 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-cd6d01\"/\"openshift-service-ca.crt\"" Nov 03 10:53:29.084748 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.084718 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-home\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.084748 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.084753 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-dshm\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.084984 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.084865 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-tmp-dir\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.084984 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.084917 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mck68\" (UniqueName: \"kubernetes.io/projected/9d332a74-4865-489a-8fca-4dfa1ac23970-kube-api-access-mck68\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.085059 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.085011 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d332a74-4865-489a-8fca-4dfa1ac23970-tls-certs\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.085099 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.085070 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-model-cache\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.187364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.186535 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-home\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.187364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.186584 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-dshm\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.187364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.186644 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-tmp-dir\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.187364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.186702 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mck68\" (UniqueName: \"kubernetes.io/projected/9d332a74-4865-489a-8fca-4dfa1ac23970-kube-api-access-mck68\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.187364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.186757 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d332a74-4865-489a-8fca-4dfa1ac23970-tls-certs\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.187364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.186809 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-model-cache\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.187364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.187242 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-model-cache\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.188113 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.187550 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-home\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.188342 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.188280 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-tmp-dir\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.190635 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.190612 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-dshm\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.197282 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.197228 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d332a74-4865-489a-8fca-4dfa1ac23970-tls-certs\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.200179 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.200126 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mck68\" (UniqueName: \"kubernetes.io/projected/9d332a74-4865-489a-8fca-4dfa1ac23970-kube-api-access-mck68\") pod \"praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.278924 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.278843 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:29.564626 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:29.564590 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn"] Nov 03 10:53:29.565675 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:53:29.565650 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d332a74_4865_489a_8fca_4dfa1ac23970.slice/crio-cb1b0256cd5c41e17f8fc8e844a7a40745b3ff04d284e7f5c44ab25db5bf4124 WatchSource:0}: Error finding container cb1b0256cd5c41e17f8fc8e844a7a40745b3ff04d284e7f5c44ab25db5bf4124: Status 404 returned error can't find the container with id cb1b0256cd5c41e17f8fc8e844a7a40745b3ff04d284e7f5c44ab25db5bf4124 Nov 03 10:53:30.033725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:30.033693 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" event={"ID":"9d332a74-4865-489a-8fca-4dfa1ac23970","Type":"ContainerStarted","Data":"0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2"} Nov 03 10:53:30.033885 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:30.033735 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" event={"ID":"9d332a74-4865-489a-8fca-4dfa1ac23970","Type":"ContainerStarted","Data":"cb1b0256cd5c41e17f8fc8e844a7a40745b3ff04d284e7f5c44ab25db5bf4124"} Nov 03 10:53:30.058068 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:30.058011 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" podStartSLOduration=2.057995911 podStartE2EDuration="2.057995911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:30.053748545 +0000 UTC m=+1488.038619832" watchObservedRunningTime="2026-11-03 10:53:30.057995911 +0000 UTC m=+1488.042867208" Nov 03 10:53:30.279825 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:30.279789 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:35.747453 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.747413 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl"] Nov 03 10:53:35.753111 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.753068 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.767625 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.767595 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl"] Nov 03 10:53:35.768018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.767997 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-18264e-serving-cert\"" Nov 03 10:53:35.806536 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.806499 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-metrics-certs\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.806742 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.806547 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c69512-a69f-4421-8372-2dbda197bcdf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.806833 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.806816 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5rb\" (UniqueName: \"kubernetes.io/projected/a4c69512-a69f-4421-8372-2dbda197bcdf-kube-api-access-4r5rb\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.806901 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.806854 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.909456 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.909409 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5rb\" (UniqueName: \"kubernetes.io/projected/a4c69512-a69f-4421-8372-2dbda197bcdf-kube-api-access-4r5rb\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.910197 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.910168 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.910533 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:35.910493 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-serving-cert: secret "maas-api-e2e-ipp-a-18264e-serving-cert" not found Nov 03 10:53:35.910643 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:35.910622 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls podName:a4c69512-a69f-4421-8372-2dbda197bcdf nodeName:}" failed. No retries permitted until 2026-11-03 10:53:36.410566824 +0000 UTC m=+1494.395438541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls") pod "maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" (UID: "a4c69512-a69f-4421-8372-2dbda197bcdf") : secret "maas-api-e2e-ipp-a-18264e-serving-cert" not found Nov 03 10:53:35.911884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.911858 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-metrics-certs\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.912056 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.912036 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c69512-a69f-4421-8372-2dbda197bcdf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.913662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.913601 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c69512-a69f-4421-8372-2dbda197bcdf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.918008 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.917549 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-metrics-certs\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:35.920375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:35.920350 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5rb\" (UniqueName: \"kubernetes.io/projected/a4c69512-a69f-4421-8372-2dbda197bcdf-kube-api-access-4r5rb\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:36.420358 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:36.420314 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:36.465979 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:36.465939 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:36.518634 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:36.518592 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh"] Nov 03 10:53:36.518881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:36.518851 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" podUID="679be1a2-3c56-467f-940c-fac577127482" containerName="payload-processing" containerID="cri-o://88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc" gracePeriod=30 Nov 03 10:53:36.534610 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:36.534577 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr"] Nov 03 10:53:36.535176 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:36.535145 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" podUID="4b44b98a-26df-4b24-a982-bb77a7bc12d4" containerName="payload-pre-processing" containerID="cri-o://a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce" gracePeriod=30 Nov 03 10:53:36.684602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:36.684517 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:36.956743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:36.956706 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:53:37.030282 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.030249 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679be1a2-3c56-467f-940c-fac577127482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679be1a2-3c56-467f-940c-fac577127482-plugins-config-volume\") pod \"679be1a2-3c56-467f-940c-fac577127482\" (UID: \"679be1a2-3c56-467f-940c-fac577127482\") " Nov 03 10:53:37.030482 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.030312 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679be1a2-3c56-467f-940c-fac577127482-kube-api-access-tf69m\" (UniqueName: \"kubernetes.io/projected/679be1a2-3c56-467f-940c-fac577127482-kube-api-access-tf69m\") pod \"679be1a2-3c56-467f-940c-fac577127482\" (UID: \"679be1a2-3c56-467f-940c-fac577127482\") " Nov 03 10:53:37.030914 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.030873 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679be1a2-3c56-467f-940c-fac577127482-plugins-config-volume" pod "679be1a2-3c56-467f-940c-fac577127482" (UID: "679be1a2-3c56-467f-940c-fac577127482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:53:37.031066 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.031043 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679be1a2-3c56-467f-940c-fac577127482-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:37.033246 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.033220 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679be1a2-3c56-467f-940c-fac577127482-kube-api-access-tf69m" pod "679be1a2-3c56-467f-940c-fac577127482" (UID: "679be1a2-3c56-467f-940c-fac577127482"). InnerVolumeSpecName "kube-api-access-tf69m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:53:37.065040 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.065012 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:53:37.097666 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.097628 5086 generic.go:356] "Generic (PLEG): container finished" podID="4b44b98a-26df-4b24-a982-bb77a7bc12d4" containerID="a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce" exitCode=0 Nov 03 10:53:37.097832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.097677 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" Nov 03 10:53:37.097832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.097689 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" event={"ID":"4b44b98a-26df-4b24-a982-bb77a7bc12d4","Type":"ContainerDied","Data":"a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce"} Nov 03 10:53:37.097832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.097745 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr" event={"ID":"4b44b98a-26df-4b24-a982-bb77a7bc12d4","Type":"ContainerDied","Data":"447b41b1c45b156d96618ac62a2f8fd5cfeede674441b8ef197d868b0ad5773e"} Nov 03 10:53:37.097832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.097775 5086 scope.go:122] "RemoveContainer" containerID="a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce" Nov 03 10:53:37.099309 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.099279 5086 generic.go:356] "Generic (PLEG): container finished" podID="679be1a2-3c56-467f-940c-fac577127482" containerID="88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc" exitCode=0 Nov 03 10:53:37.099428 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.099410 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" event={"ID":"679be1a2-3c56-467f-940c-fac577127482","Type":"ContainerDied","Data":"88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc"} Nov 03 10:53:37.099504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.099485 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" event={"ID":"679be1a2-3c56-467f-940c-fac577127482","Type":"ContainerDied","Data":"58a9654963eb04fddd2fee2f41a775e8e82a1270be1bdee8fe4036a66826497e"} Nov 03 10:53:37.099610 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.099594 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh" Nov 03 10:53:37.111560 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.111536 5086 scope.go:122] "RemoveContainer" containerID="a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce" Nov 03 10:53:37.113637 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:37.113598 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce\": container with ID starting with a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce not found: ID does not exist" containerID="a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce" Nov 03 10:53:37.113750 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.113639 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce"} err="failed to get container status \"a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce\": rpc error: code = NotFound desc = could not find container \"a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce\": container with ID starting with a2eb80f3aaa996a5708a243321d0b0742560c8ee7366127dadf76f4a65f0b8ce not found: ID does not exist" Nov 03 10:53:37.113750 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.113663 5086 scope.go:122] "RemoveContainer" containerID="88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc" Nov 03 10:53:37.126559 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.126474 5086 scope.go:122] "RemoveContainer" containerID="88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc" Nov 03 10:53:37.126783 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:37.126765 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc\": container with ID starting with 88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc not found: ID does not exist" containerID="88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc" Nov 03 10:53:37.126862 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.126795 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc"} err="failed to get container status \"88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc\": rpc error: code = NotFound desc = could not find container \"88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc\": container with ID starting with 88001c6eb23f547285a1e8a9114a4039e76de8aa543093bb8449a2bbb99441bc not found: ID does not exist" Nov 03 10:53:37.128171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.128154 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh"] Nov 03 10:53:37.131894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.131868 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-5555c8f5d6-wnknh"] Nov 03 10:53:37.132362 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.132284 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b44b98a-26df-4b24-a982-bb77a7bc12d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b44b98a-26df-4b24-a982-bb77a7bc12d4-plugins-config-volume\") pod \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\" (UID: \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\") " Nov 03 10:53:37.132485 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.132471 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b44b98a-26df-4b24-a982-bb77a7bc12d4-kube-api-access-mmfhj\" (UniqueName: \"kubernetes.io/projected/4b44b98a-26df-4b24-a982-bb77a7bc12d4-kube-api-access-mmfhj\") pod \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\" (UID: \"4b44b98a-26df-4b24-a982-bb77a7bc12d4\") " Nov 03 10:53:37.132632 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.132609 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b44b98a-26df-4b24-a982-bb77a7bc12d4-plugins-config-volume" pod "4b44b98a-26df-4b24-a982-bb77a7bc12d4" (UID: "4b44b98a-26df-4b24-a982-bb77a7bc12d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:53:37.132946 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.132930 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b44b98a-26df-4b24-a982-bb77a7bc12d4-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:37.132986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.132951 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf69m\" (UniqueName: \"kubernetes.io/projected/679be1a2-3c56-467f-940c-fac577127482-kube-api-access-tf69m\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:37.134612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.134588 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b44b98a-26df-4b24-a982-bb77a7bc12d4-kube-api-access-mmfhj" pod "4b44b98a-26df-4b24-a982-bb77a7bc12d4" (UID: "4b44b98a-26df-4b24-a982-bb77a7bc12d4"). InnerVolumeSpecName "kube-api-access-mmfhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:53:37.140721 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.140504 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl"] Nov 03 10:53:37.235003 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.234921 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmfhj\" (UniqueName: \"kubernetes.io/projected/4b44b98a-26df-4b24-a982-bb77a7bc12d4-kube-api-access-mmfhj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:53:37.423081 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.423045 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr"] Nov 03 10:53:37.426113 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:37.426090 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa420c-bbd7c4b4c-sn4wr"] Nov 03 10:53:38.110388 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:38.110349 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" event={"ID":"a4c69512-a69f-4421-8372-2dbda197bcdf","Type":"ContainerStarted","Data":"30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22"} Nov 03 10:53:38.110832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:38.110411 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" event={"ID":"a4c69512-a69f-4421-8372-2dbda197bcdf","Type":"ContainerStarted","Data":"f95e91ab6e7e1bb02d2d96a47c0f5f500613929331baedf7b470ee522d206d25"} Nov 03 10:53:38.110832 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:38.110549 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:38.131079 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:38.131012 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" podStartSLOduration=2.602085024 podStartE2EDuration="3.130993642s" totalImagesPullingTime="528.908618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:38.127940179 +0000 UTC m=+1496.112811463" watchObservedRunningTime="2026-11-03 10:53:38.130993642 +0000 UTC m=+1496.115864926" Nov 03 10:53:38.180629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:38.180594 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b44b98a-26df-4b24-a982-bb77a7bc12d4" path="/var/lib/kubelet/pods/4b44b98a-26df-4b24-a982-bb77a7bc12d4/volumes" Nov 03 10:53:38.181030 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:38.181012 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679be1a2-3c56-467f-940c-fac577127482" path="/var/lib/kubelet/pods/679be1a2-3c56-467f-940c-fac577127482/volumes" Nov 03 10:53:39.279649 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:39.279614 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:39.292586 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:39.292556 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:40.139407 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:40.139373 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:53:42.825441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:42.825410 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:53:42.832492 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:42.832470 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:53:42.869602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:42.869571 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:53:42.885330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:42.876695 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:53:44.138357 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:44.135468 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:53:54.022187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.022153 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg"] Nov 03 10:53:54.022866 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.022830 5086 cpu_manager.go:414] "containerMap: removing container" podUID="679be1a2-3c56-467f-940c-fac577127482" containerName="payload-processing" Nov 03 10:53:54.022866 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.022844 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679be1a2-3c56-467f-940c-fac577127482" containerName="payload-processing" Nov 03 10:53:54.023001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.022875 5086 cpu_manager.go:414] "containerMap: removing container" podUID="4b44b98a-26df-4b24-a982-bb77a7bc12d4" containerName="payload-pre-processing" Nov 03 10:53:54.023001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.022881 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b44b98a-26df-4b24-a982-bb77a7bc12d4" containerName="payload-pre-processing" Nov 03 10:53:54.023001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.022971 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="679be1a2-3c56-467f-940c-fac577127482" containerName="payload-processing" Nov 03 10:53:54.023001 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.022985 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b44b98a-26df-4b24-a982-bb77a7bc12d4" containerName="payload-pre-processing" Nov 03 10:53:54.028502 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.028479 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:53:54.033119 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.033093 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-18264e\"" Nov 03 10:53:54.039763 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.037770 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg"] Nov 03 10:53:54.122318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.099878 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-18264e-dockercfg-bmlpj\"" Nov 03 10:53:54.146974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.141277 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xblzh\" (UniqueName: \"kubernetes.io/projected/d76221c7-dd2e-4ac4-9991-5b02a58f222c-kube-api-access-xblzh\") pod \"payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg\" (UID: \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:53:54.146974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.141483 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76221c7-dd2e-4ac4-9991-5b02a58f222c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg\" (UID: \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:53:54.243613 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.243570 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xblzh\" (UniqueName: \"kubernetes.io/projected/d76221c7-dd2e-4ac4-9991-5b02a58f222c-kube-api-access-xblzh\") pod \"payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg\" (UID: \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:53:54.243809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.243765 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76221c7-dd2e-4ac4-9991-5b02a58f222c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg\" (UID: \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:53:54.244782 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.244751 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76221c7-dd2e-4ac4-9991-5b02a58f222c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg\" (UID: \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:53:54.253075 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.253047 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xblzh\" (UniqueName: \"kubernetes.io/projected/d76221c7-dd2e-4ac4-9991-5b02a58f222c-kube-api-access-xblzh\") pod \"payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg\" (UID: \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:53:54.280538 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:53:54.280434 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:53:54.351476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.351442 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:53:54.632398 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:54.632366 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg"] Nov 03 10:53:54.632981 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:53:54.632955 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76221c7_dd2e_4ac4_9991_5b02a58f222c.slice/crio-6cb21594430c3f66163f76fd9ffb28b7cc5ba973c96723f0a79c18ac26836b6d WatchSource:0}: Error finding container 6cb21594430c3f66163f76fd9ffb28b7cc5ba973c96723f0a79c18ac26836b6d: Status 404 returned error can't find the container with id 6cb21594430c3f66163f76fd9ffb28b7cc5ba973c96723f0a79c18ac26836b6d Nov 03 10:53:55.404256 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:55.404215 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" event={"ID":"d76221c7-dd2e-4ac4-9991-5b02a58f222c","Type":"ContainerStarted","Data":"839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441"} Nov 03 10:53:55.404812 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:55.404283 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" event={"ID":"d76221c7-dd2e-4ac4-9991-5b02a58f222c","Type":"ContainerStarted","Data":"6cb21594430c3f66163f76fd9ffb28b7cc5ba973c96723f0a79c18ac26836b6d"} Nov 03 10:53:55.404812 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:55.404556 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:53:55.425704 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:55.425653 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" podStartSLOduration=1.425638435 podStartE2EDuration="1.425638435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:55.421549994 +0000 UTC m=+1513.406421278" watchObservedRunningTime="2026-11-03 10:53:55.425638435 +0000 UTC m=+1513.410509752" Nov 03 10:53:57.477964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.477926 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr"] Nov 03 10:53:57.482630 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.482598 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.491506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.491473 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-35afa6-serving-cert\"" Nov 03 10:53:57.492814 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.492788 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr"] Nov 03 10:53:57.606869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.606836 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.607062 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.606976 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-metrics-certs\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.607062 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.607011 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.607142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.607118 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zz5\" (UniqueName: \"kubernetes.io/projected/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-kube-api-access-f5zz5\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.708434 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.708396 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-metrics-certs\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.708633 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.708449 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.708633 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.708503 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zz5\" (UniqueName: \"kubernetes.io/projected/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-kube-api-access-f5zz5\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.708765 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.708631 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.709232 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.709206 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.711115 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.711086 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-metrics-certs\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.711223 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.711140 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.716061 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.716036 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zz5\" (UniqueName: \"kubernetes.io/projected/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-kube-api-access-f5zz5\") pod \"maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:57.808352 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:57.808221 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:58.099113 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:58.099078 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr"] Nov 03 10:53:58.102358 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:53:58.101637 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3cfac0_27ec_4c71_8593_8ff3201fdc8a.slice/crio-9a3b3f103d111002fb935f7e89110cf0534e001874be0651842780cfeb5330e1 WatchSource:0}: Error finding container 9a3b3f103d111002fb935f7e89110cf0534e001874be0651842780cfeb5330e1: Status 404 returned error can't find the container with id 9a3b3f103d111002fb935f7e89110cf0534e001874be0651842780cfeb5330e1 Nov 03 10:53:58.425946 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:58.425858 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" event={"ID":"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a","Type":"ContainerStarted","Data":"9a3b3f103d111002fb935f7e89110cf0534e001874be0651842780cfeb5330e1"} Nov 03 10:53:59.447195 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:59.447156 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" event={"ID":"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a","Type":"ContainerStarted","Data":"d627a86747ff0c4c81b6f54786d794287893fc5a8ffdd94a9109b92c854c8055"} Nov 03 10:53:59.447684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:59.447226 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:53:59.467629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:53:59.467567 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" podStartSLOduration=2.022208291 podStartE2EDuration="2.467548446s" totalImagesPullingTime="445.340155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:59.463083663 +0000 UTC m=+1517.447954960" watchObservedRunningTime="2026-11-03 10:53:59.467548446 +0000 UTC m=+1517.452419729" Nov 03 10:54:02.859321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:02.858388 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Nov 03 10:54:03.012320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:02.997318 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 10:54:03.177165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:03.176369 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 10:54:05.481323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:05.472420 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:54:06.414321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:06.414277 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:54:18.159948 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:18.159903 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz"] Nov 03 10:54:18.160418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:18.160140 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" containerID="cri-o://196282c1b29cbc0d33d870975594f7ee7df6738e8218b85c0acda5fac6e76f30" gracePeriod=30 Nov 03 10:54:22.558642 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.558602 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj"] Nov 03 10:54:22.565054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.565027 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.579591 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.579561 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8bbb6e-serving-cert\"" Nov 03 10:54:22.588771 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.588739 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj"] Nov 03 10:54:22.685727 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.685693 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.685919 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.685860 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-metrics-certs\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.685919 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.685907 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhbs\" (UniqueName: \"kubernetes.io/projected/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-kube-api-access-pjhbs\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.686077 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.686054 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.787850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.787812 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.788067 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.787923 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.788067 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.788033 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-metrics-certs\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.788215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.788096 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhbs\" (UniqueName: \"kubernetes.io/projected/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-kube-api-access-pjhbs\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.788751 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.788728 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.791040 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.791015 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.794589 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.794557 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-metrics-certs\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.796919 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.796897 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhbs\" (UniqueName: \"kubernetes.io/projected/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-kube-api-access-pjhbs\") pod \"maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:22.893990 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:22.893952 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:23.249976 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:23.248424 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj"] Nov 03 10:54:23.269782 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:54:23.269745 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e622f9f_0bb3_4e1f_b1b8_f4c05d81e795.slice/crio-3b00583fad095a8da9f46141d3568cd12ca74affabcdd60e1d106765b2b2a46f WatchSource:0}: Error finding container 3b00583fad095a8da9f46141d3568cd12ca74affabcdd60e1d106765b2b2a46f: Status 404 returned error can't find the container with id 3b00583fad095a8da9f46141d3568cd12ca74affabcdd60e1d106765b2b2a46f Nov 03 10:54:23.835250 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:23.835207 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" event={"ID":"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795","Type":"ContainerStarted","Data":"3b00583fad095a8da9f46141d3568cd12ca74affabcdd60e1d106765b2b2a46f"} Nov 03 10:54:23.840211 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:23.840177 5086 generic.go:356] "Generic (PLEG): container finished" podID="194ed661-f22d-48d6-ba45-016943dee4b0" containerID="196282c1b29cbc0d33d870975594f7ee7df6738e8218b85c0acda5fac6e76f30" exitCode=0 Nov 03 10:54:23.840394 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:23.840255 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" event={"ID":"194ed661-f22d-48d6-ba45-016943dee4b0","Type":"ContainerDied","Data":"196282c1b29cbc0d33d870975594f7ee7df6738e8218b85c0acda5fac6e76f30"} Nov 03 10:54:23.963406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:23.956375 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.042985 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/194ed661-f22d-48d6-ba45-016943dee4b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194ed661-f22d-48d6-ba45-016943dee4b0-istio-podinfo\") pod \"194ed661-f22d-48d6-ba45-016943dee4b0\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.043042 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194ed661-f22d-48d6-ba45-016943dee4b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194ed661-f22d-48d6-ba45-016943dee4b0-istiod-ca-cert\") pod \"194ed661-f22d-48d6-ba45-016943dee4b0\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.043076 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-envoy\") pod \"194ed661-f22d-48d6-ba45-016943dee4b0\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.043209 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-kube-api-access-f6bpm\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-kube-api-access-f6bpm\") pod \"194ed661-f22d-48d6-ba45-016943dee4b0\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.043290 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-credential-socket\") pod \"194ed661-f22d-48d6-ba45-016943dee4b0\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.043386 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-istio-token\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-istio-token\") pod \"194ed661-f22d-48d6-ba45-016943dee4b0\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.043483 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-data\") pod \"194ed661-f22d-48d6-ba45-016943dee4b0\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.043518 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-socket\") pod \"194ed661-f22d-48d6-ba45-016943dee4b0\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.043627 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-certs\") pod \"194ed661-f22d-48d6-ba45-016943dee4b0\" (UID: \"194ed661-f22d-48d6-ba45-016943dee4b0\") " Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.044932 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-certs" pod "194ed661-f22d-48d6-ba45-016943dee4b0" (UID: "194ed661-f22d-48d6-ba45-016943dee4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.045125 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-credential-socket" pod "194ed661-f22d-48d6-ba45-016943dee4b0" (UID: "194ed661-f22d-48d6-ba45-016943dee4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:24.046314 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.045702 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194ed661-f22d-48d6-ba45-016943dee4b0-istiod-ca-cert" pod "194ed661-f22d-48d6-ba45-016943dee4b0" (UID: "194ed661-f22d-48d6-ba45-016943dee4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:54:24.053435 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.051822 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-socket" pod "194ed661-f22d-48d6-ba45-016943dee4b0" (UID: "194ed661-f22d-48d6-ba45-016943dee4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:24.053435 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.052043 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-data" pod "194ed661-f22d-48d6-ba45-016943dee4b0" (UID: "194ed661-f22d-48d6-ba45-016943dee4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:24.055803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.055764 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-envoy" pod "194ed661-f22d-48d6-ba45-016943dee4b0" (UID: "194ed661-f22d-48d6-ba45-016943dee4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:24.059498 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.059451 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-istio-token" pod "194ed661-f22d-48d6-ba45-016943dee4b0" (UID: "194ed661-f22d-48d6-ba45-016943dee4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:54:24.061360 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.061309 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/194ed661-f22d-48d6-ba45-016943dee4b0-istio-podinfo" pod "194ed661-f22d-48d6-ba45-016943dee4b0" (UID: "194ed661-f22d-48d6-ba45-016943dee4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:54:24.062165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.062136 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-kube-api-access-f6bpm" pod "194ed661-f22d-48d6-ba45-016943dee4b0" (UID: "194ed661-f22d-48d6-ba45-016943dee4b0"). InnerVolumeSpecName "kube-api-access-f6bpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:54:24.154323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.153169 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6bpm\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-kube-api-access-f6bpm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:24.154323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.153208 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:24.154323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.153225 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194ed661-f22d-48d6-ba45-016943dee4b0-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:24.154323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.153240 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:24.154323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.153255 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:24.154323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.153270 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:24.154323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.153286 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194ed661-f22d-48d6-ba45-016943dee4b0-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:24.154323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.153318 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194ed661-f22d-48d6-ba45-016943dee4b0-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:24.154323 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.153335 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194ed661-f22d-48d6-ba45-016943dee4b0-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:24.294912 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:54:24.294806 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:54:24.883319 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.882618 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" event={"ID":"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795","Type":"ContainerStarted","Data":"20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea"} Nov 03 10:54:24.883808 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.883726 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:24.905320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.895581 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" event={"ID":"194ed661-f22d-48d6-ba45-016943dee4b0","Type":"ContainerDied","Data":"b4bb7c1d682ac6f89d050df63443ccc9d82f43b3d4faea591b8f5bf7052f759c"} Nov 03 10:54:24.905320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.895654 5086 scope.go:122] "RemoveContainer" containerID="196282c1b29cbc0d33d870975594f7ee7df6738e8218b85c0acda5fac6e76f30" Nov 03 10:54:24.905320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.895894 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz" Nov 03 10:54:24.909315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:24.906441 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" podStartSLOduration=2.424332209 podStartE2EDuration="2.906422799s" totalImagesPullingTime="482.09059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:24.905316597 +0000 UTC m=+1542.890187874" watchObservedRunningTime="2026-11-03 10:54:24.906422799 +0000 UTC m=+1542.891294073" Nov 03 10:54:25.152380 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:25.148363 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz"] Nov 03 10:54:25.233989 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:25.233950 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-zb6jz"] Nov 03 10:54:26.273140 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:26.273099 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" path="/var/lib/kubelet/pods/194ed661-f22d-48d6-ba45-016943dee4b0/volumes" Nov 03 10:54:28.385875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:28.385835 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72"] Nov 03 10:54:28.386418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:28.386044 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" containerID="cri-o://bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7" gracePeriod=30 Nov 03 10:54:31.967603 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:31.967574 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:54:33.671959 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.671928 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:54:33.762907 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.762873 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-podinfo\") pod \"44b9885d-8b0e-4e0b-9628-30034fb60dda\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " Nov 03 10:54:33.763116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.762942 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-envoy\") pod \"44b9885d-8b0e-4e0b-9628-30034fb60dda\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " Nov 03 10:54:33.763116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763017 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-kube-api-access-2zd6j\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-kube-api-access-2zd6j\") pod \"44b9885d-8b0e-4e0b-9628-30034fb60dda\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " Nov 03 10:54:33.763116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763063 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-socket\") pod \"44b9885d-8b0e-4e0b-9628-30034fb60dda\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " Nov 03 10:54:33.763116 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763108 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b9885d-8b0e-4e0b-9628-30034fb60dda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b9885d-8b0e-4e0b-9628-30034fb60dda-istiod-ca-cert\") pod \"44b9885d-8b0e-4e0b-9628-30034fb60dda\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " Nov 03 10:54:33.763386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763244 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-data\") pod \"44b9885d-8b0e-4e0b-9628-30034fb60dda\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " Nov 03 10:54:33.763386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763337 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-credential-socket\") pod \"44b9885d-8b0e-4e0b-9628-30034fb60dda\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " Nov 03 10:54:33.763386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763381 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-token\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-token\") pod \"44b9885d-8b0e-4e0b-9628-30034fb60dda\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " Nov 03 10:54:33.763538 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763418 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-socket" pod "44b9885d-8b0e-4e0b-9628-30034fb60dda" (UID: "44b9885d-8b0e-4e0b-9628-30034fb60dda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:33.763538 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763517 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-certs\") pod \"44b9885d-8b0e-4e0b-9628-30034fb60dda\" (UID: \"44b9885d-8b0e-4e0b-9628-30034fb60dda\") " Nov 03 10:54:33.763655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763569 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-credential-socket" pod "44b9885d-8b0e-4e0b-9628-30034fb60dda" (UID: "44b9885d-8b0e-4e0b-9628-30034fb60dda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:33.763655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763581 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-data" pod "44b9885d-8b0e-4e0b-9628-30034fb60dda" (UID: "44b9885d-8b0e-4e0b-9628-30034fb60dda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:33.763655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763604 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b9885d-8b0e-4e0b-9628-30034fb60dda-istiod-ca-cert" pod "44b9885d-8b0e-4e0b-9628-30034fb60dda" (UID: "44b9885d-8b0e-4e0b-9628-30034fb60dda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:54:33.763911 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.763860 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-certs" pod "44b9885d-8b0e-4e0b-9628-30034fb60dda" (UID: "44b9885d-8b0e-4e0b-9628-30034fb60dda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:33.764071 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.764051 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:33.764160 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.764072 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b9885d-8b0e-4e0b-9628-30034fb60dda-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:33.764160 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.764082 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:33.764160 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.764091 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:33.764160 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.764110 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:33.765492 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.765467 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-kube-api-access-2zd6j" pod "44b9885d-8b0e-4e0b-9628-30034fb60dda" (UID: "44b9885d-8b0e-4e0b-9628-30034fb60dda"). InnerVolumeSpecName "kube-api-access-2zd6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:54:33.765600 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.765508 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-envoy" pod "44b9885d-8b0e-4e0b-9628-30034fb60dda" (UID: "44b9885d-8b0e-4e0b-9628-30034fb60dda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:54:33.765689 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.765663 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-podinfo" pod "44b9885d-8b0e-4e0b-9628-30034fb60dda" (UID: "44b9885d-8b0e-4e0b-9628-30034fb60dda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:54:33.765754 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.765739 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-token" pod "44b9885d-8b0e-4e0b-9628-30034fb60dda" (UID: "44b9885d-8b0e-4e0b-9628-30034fb60dda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:54:33.865720 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.865684 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:33.865720 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.865715 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:33.865720 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.865726 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b9885d-8b0e-4e0b-9628-30034fb60dda-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:33.866019 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:33.865735 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zd6j\" (UniqueName: \"kubernetes.io/projected/44b9885d-8b0e-4e0b-9628-30034fb60dda-kube-api-access-2zd6j\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:54:34.019688 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.019605 5086 generic.go:356] "Generic (PLEG): container finished" podID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerID="bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7" exitCode=0 Nov 03 10:54:34.019688 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.019656 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" Nov 03 10:54:34.019893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.019688 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" event={"ID":"44b9885d-8b0e-4e0b-9628-30034fb60dda","Type":"ContainerDied","Data":"bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7"} Nov 03 10:54:34.019893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.019748 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72" event={"ID":"44b9885d-8b0e-4e0b-9628-30034fb60dda","Type":"ContainerDied","Data":"254b7b28a6a43d9b726faa49afec376445dc915bce027c770a87cad4ded0ba3b"} Nov 03 10:54:34.019893 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.019778 5086 scope.go:122] "RemoveContainer" containerID="bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7" Nov 03 10:54:34.031207 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.031189 5086 scope.go:122] "RemoveContainer" containerID="bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7" Nov 03 10:54:34.031549 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:54:34.031529 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7\": container with ID starting with bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7 not found: ID does not exist" containerID="bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7" Nov 03 10:54:34.031611 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.031560 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7"} err="failed to get container status \"bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7\": rpc error: code = NotFound desc = could not find container \"bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7\": container with ID starting with bed2849966522f335550926ac38e04bf6fd5c568294fac1a9a753337a90792b7 not found: ID does not exist" Nov 03 10:54:34.046211 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.046188 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72"] Nov 03 10:54:34.049325 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.049271 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-gkh72"] Nov 03 10:54:34.171973 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:34.171941 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" path="/var/lib/kubelet/pods/44b9885d-8b0e-4e0b-9628-30034fb60dda/volumes" Nov 03 10:54:37.565759 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.565720 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr"] Nov 03 10:54:37.566733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.566710 5086 cpu_manager.go:414] "containerMap: removing container" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" Nov 03 10:54:37.566733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.566732 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" Nov 03 10:54:37.566931 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.566756 5086 cpu_manager.go:414] "containerMap: removing container" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" Nov 03 10:54:37.566931 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.566764 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" Nov 03 10:54:37.566931 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.566927 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="194ed661-f22d-48d6-ba45-016943dee4b0" containerName="istio-proxy" Nov 03 10:54:37.567109 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.566941 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b9885d-8b0e-4e0b-9628-30034fb60dda" containerName="istio-proxy" Nov 03 10:54:37.571179 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.571158 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.587538 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.587504 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-343e30-serving-cert\"" Nov 03 10:54:37.593905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.593871 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr"] Nov 03 10:54:37.618486 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.618447 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-metrics-certs\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.618670 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.618501 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.618670 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.618654 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6wm\" (UniqueName: \"kubernetes.io/projected/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-kube-api-access-8t6wm\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.618793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.618709 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.720590 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.720550 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6wm\" (UniqueName: \"kubernetes.io/projected/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-kube-api-access-8t6wm\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.720766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.720635 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.720856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.720797 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-metrics-certs\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.720856 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.720829 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.720991 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:54:37.720975 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-serving-cert: secret "maas-api-e2e-praxis-b-343e30-serving-cert" not found Nov 03 10:54:37.721070 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:54:37.721057 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls podName:de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4 nodeName:}" failed. No retries permitted until 2026-11-03 10:54:38.221034472 +0000 UTC m=+1556.205905738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls") pod "maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" (UID: "de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4") : secret "maas-api-e2e-praxis-b-343e30-serving-cert" not found Nov 03 10:54:37.721860 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.721832 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.725731 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.725707 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-metrics-certs\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:37.732841 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:37.732813 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6wm\" (UniqueName: \"kubernetes.io/projected/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-kube-api-access-8t6wm\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:38.227646 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:38.227602 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:38.231850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:38.231818 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:38.507005 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:38.506923 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:38.788046 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:54:38.788013 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2f0e5f_a3f0_4fb9_a8b4_9427ced52bf4.slice/crio-848484513bca93beffb54732a862d0174edd3629bb0318e64eb487470fe602d0 WatchSource:0}: Error finding container 848484513bca93beffb54732a862d0174edd3629bb0318e64eb487470fe602d0: Status 404 returned error can't find the container with id 848484513bca93beffb54732a862d0174edd3629bb0318e64eb487470fe602d0 Nov 03 10:54:38.788956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:38.788851 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr"] Nov 03 10:54:39.057813 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:39.057724 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" event={"ID":"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4","Type":"ContainerStarted","Data":"848484513bca93beffb54732a862d0174edd3629bb0318e64eb487470fe602d0"} Nov 03 10:54:40.066765 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:40.066726 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" event={"ID":"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4","Type":"ContainerStarted","Data":"3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108"} Nov 03 10:54:40.067282 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:40.066973 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:40.089018 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:40.088951 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" podStartSLOduration=2.719712844 podStartE2EDuration="3.088930369s" totalImagesPullingTime="369.217525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:40.083698217 +0000 UTC m=+1558.068569501" watchObservedRunningTime="2026-11-03 10:54:40.088930369 +0000 UTC m=+1558.073801652" Nov 03 10:54:45.700106 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.700052 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6"] Nov 03 10:54:45.705804 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.705779 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:54:45.709445 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.709404 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8bbb6e\"" Nov 03 10:54:45.712636 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.712613 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6"] Nov 03 10:54:45.735848 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.735822 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6"] Nov 03 10:54:45.740346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.740328 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:45.747163 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.747139 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8bbb6e-dockercfg-9cgb8\"" Nov 03 10:54:45.752656 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.752631 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6"] Nov 03 10:54:45.868098 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.847353 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh59b\" (UniqueName: \"kubernetes.io/projected/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-kube-api-access-hh59b\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6\" (UID: \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:54:45.868098 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.847421 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6\" (UID: \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:45.868098 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.847546 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgf4\" (UniqueName: \"kubernetes.io/projected/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-kube-api-access-rtgf4\") pod \"payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6\" (UID: \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:45.868098 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.847604 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6\" (UID: \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:54:45.948803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.948766 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh59b\" (UniqueName: \"kubernetes.io/projected/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-kube-api-access-hh59b\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6\" (UID: \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:54:45.948803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.948811 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6\" (UID: \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:45.949089 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.948883 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgf4\" (UniqueName: \"kubernetes.io/projected/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-kube-api-access-rtgf4\") pod \"payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6\" (UID: \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:45.949089 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.948916 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6\" (UID: \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:54:45.949752 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.949727 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6\" (UID: \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:54:45.949923 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.949904 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6\" (UID: \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:45.957234 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.957171 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh59b\" (UniqueName: \"kubernetes.io/projected/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-kube-api-access-hh59b\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6\" (UID: \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:54:45.957385 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:45.957313 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgf4\" (UniqueName: \"kubernetes.io/projected/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-kube-api-access-rtgf4\") pod \"payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6\" (UID: \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:46.024596 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:46.024564 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:54:46.057370 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:46.057144 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:46.085095 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:46.085065 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:54:46.316094 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:46.316048 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6"] Nov 03 10:54:46.316773 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:54:46.316745 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3741a6c0_d3e8_4f36_aed0_c4f3a5fb62e1.slice/crio-a15c9a228caf8cdac8510bc802a1220173ed798c946a4f899cf2b8c3ffe3b2c7 WatchSource:0}: Error finding container a15c9a228caf8cdac8510bc802a1220173ed798c946a4f899cf2b8c3ffe3b2c7: Status 404 returned error can't find the container with id a15c9a228caf8cdac8510bc802a1220173ed798c946a4f899cf2b8c3ffe3b2c7 Nov 03 10:54:46.360718 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:46.360696 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6"] Nov 03 10:54:46.361829 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:54:46.361805 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaff81c_2bc5_4bd6_bd9a_3179f11e1a95.slice/crio-3347ea9f7ed223a5160fd2cd3f849d69df577cf2303d332bf64597523ed0412f WatchSource:0}: Error finding container 3347ea9f7ed223a5160fd2cd3f849d69df577cf2303d332bf64597523ed0412f: Status 404 returned error can't find the container with id 3347ea9f7ed223a5160fd2cd3f849d69df577cf2303d332bf64597523ed0412f Nov 03 10:54:47.134549 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:47.134494 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" event={"ID":"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1","Type":"ContainerStarted","Data":"db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba"} Nov 03 10:54:47.135047 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:47.134593 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:54:47.135047 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:47.134617 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" event={"ID":"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1","Type":"ContainerStarted","Data":"a15c9a228caf8cdac8510bc802a1220173ed798c946a4f899cf2b8c3ffe3b2c7"} Nov 03 10:54:47.136521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:47.136493 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" event={"ID":"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95","Type":"ContainerStarted","Data":"1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1"} Nov 03 10:54:47.136647 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:47.136537 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" event={"ID":"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95","Type":"ContainerStarted","Data":"3347ea9f7ed223a5160fd2cd3f849d69df577cf2303d332bf64597523ed0412f"} Nov 03 10:54:47.136753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:47.136732 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:47.179820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:47.177699 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" podStartSLOduration=2.177680924 podStartE2EDuration="2.177680924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:47.158234105 +0000 UTC m=+1565.143105388" watchObservedRunningTime="2026-11-03 10:54:47.177680924 +0000 UTC m=+1565.162552208" Nov 03 10:54:47.179820 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:47.177820 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" podStartSLOduration=2.177809982 podStartE2EDuration="2.177809982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:47.175096408 +0000 UTC m=+1565.159967692" watchObservedRunningTime="2026-11-03 10:54:47.177809982 +0000 UTC m=+1565.162681267" Nov 03 10:54:50.335351 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:50.289567 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" podUID="9d332a74-4865-489a-8fca-4dfa1ac23970" containerName="main" probeResult="failure" output="Get \"https://10.132.0.147:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 10:54:54.276207 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:54:54.276131 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:54:58.153307 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:58.153223 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:54:58.153920 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:54:58.153342 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:55:06.309055 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.309008 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr"] Nov 03 10:55:06.309618 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.309261 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" podUID="bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" containerName="maas-api" containerID="cri-o://d627a86747ff0c4c81b6f54786d794287893fc5a8ffdd94a9109b92c854c8055" gracePeriod=30 Nov 03 10:55:06.489889 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.489854 5086 generic.go:356] "Generic (PLEG): container finished" podID="bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" containerID="d627a86747ff0c4c81b6f54786d794287893fc5a8ffdd94a9109b92c854c8055" exitCode=0 Nov 03 10:55:06.490043 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.489933 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" event={"ID":"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a","Type":"ContainerDied","Data":"d627a86747ff0c4c81b6f54786d794287893fc5a8ffdd94a9109b92c854c8055"} Nov 03 10:55:06.676603 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.676570 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:55:06.848504 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.848469 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-metrics-certs\") pod \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " Nov 03 10:55:06.848710 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.848595 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-kube-api-access-f5zz5\" (UniqueName: \"kubernetes.io/projected/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-kube-api-access-f5zz5\") pod \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " Nov 03 10:55:06.848810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.848787 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-maas-api-tls\") pod \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " Nov 03 10:55:06.848964 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.848839 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-openshift-service-ca\") pod \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\" (UID: \"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a\") " Nov 03 10:55:06.849266 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.849238 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-openshift-service-ca" pod "bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" (UID: "bb3cfac0-27ec-4c71-8593-8ff3201fdc8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:55:06.849457 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.849441 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:06.850723 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.850694 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-metrics-certs" pod "bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" (UID: "bb3cfac0-27ec-4c71-8593-8ff3201fdc8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:06.850806 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.850736 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-maas-api-tls" pod "bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" (UID: "bb3cfac0-27ec-4c71-8593-8ff3201fdc8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:06.850852 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.850801 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-kube-api-access-f5zz5" pod "bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" (UID: "bb3cfac0-27ec-4c71-8593-8ff3201fdc8a"). InnerVolumeSpecName "kube-api-access-f5zz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:55:06.950791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.950703 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:06.950791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.950735 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:06.950791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:06.950745 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5zz5\" (UniqueName: \"kubernetes.io/projected/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a-kube-api-access-f5zz5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:07.503981 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:07.503935 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" event={"ID":"bb3cfac0-27ec-4c71-8593-8ff3201fdc8a","Type":"ContainerDied","Data":"9a3b3f103d111002fb935f7e89110cf0534e001874be0651842780cfeb5330e1"} Nov 03 10:55:07.504547 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:07.503995 5086 scope.go:122] "RemoveContainer" containerID="d627a86747ff0c4c81b6f54786d794287893fc5a8ffdd94a9109b92c854c8055" Nov 03 10:55:07.504547 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:07.504139 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr" Nov 03 10:55:07.535430 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:07.535398 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr"] Nov 03 10:55:07.547138 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:07.546735 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-35afa6-645cc48558-fprbr"] Nov 03 10:55:08.172330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:08.172280 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" path="/var/lib/kubelet/pods/bb3cfac0-27ec-4c71-8593-8ff3201fdc8a/volumes" Nov 03 10:55:15.443241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:15.443211 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6_ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95/payload-processing/0.log" Nov 03 10:55:24.275149 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:55:24.275112 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:55:28.609065 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:28.609008 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl"] Nov 03 10:55:28.609650 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:28.609324 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" podUID="a4c69512-a69f-4421-8372-2dbda197bcdf" containerName="maas-api" containerID="cri-o://30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22" gracePeriod=30 Nov 03 10:55:28.731594 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:28.731533 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg"] Nov 03 10:55:28.732144 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:28.732086 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" podUID="d76221c7-dd2e-4ac4-9991-5b02a58f222c" containerName="payload-processing" containerID="cri-o://839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441" gracePeriod=30 Nov 03 10:55:28.956520 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:28.956481 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:55:29.112881 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.112517 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:55:29.128585 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.128554 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c69512-a69f-4421-8372-2dbda197bcdf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c69512-a69f-4421-8372-2dbda197bcdf-openshift-service-ca\") pod \"a4c69512-a69f-4421-8372-2dbda197bcdf\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " Nov 03 10:55:29.128788 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.128625 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c69512-a69f-4421-8372-2dbda197bcdf-kube-api-access-4r5rb\" (UniqueName: \"kubernetes.io/projected/a4c69512-a69f-4421-8372-2dbda197bcdf-kube-api-access-4r5rb\") pod \"a4c69512-a69f-4421-8372-2dbda197bcdf\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " Nov 03 10:55:29.128788 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.128768 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-metrics-certs\") pod \"a4c69512-a69f-4421-8372-2dbda197bcdf\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " Nov 03 10:55:29.128927 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.128903 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls\") pod \"a4c69512-a69f-4421-8372-2dbda197bcdf\" (UID: \"a4c69512-a69f-4421-8372-2dbda197bcdf\") " Nov 03 10:55:29.129005 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.128977 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c69512-a69f-4421-8372-2dbda197bcdf-openshift-service-ca" pod "a4c69512-a69f-4421-8372-2dbda197bcdf" (UID: "a4c69512-a69f-4421-8372-2dbda197bcdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:55:29.129315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.129283 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c69512-a69f-4421-8372-2dbda197bcdf-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:29.131611 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.131559 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls" pod "a4c69512-a69f-4421-8372-2dbda197bcdf" (UID: "a4c69512-a69f-4421-8372-2dbda197bcdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:29.131956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.131932 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c69512-a69f-4421-8372-2dbda197bcdf-kube-api-access-4r5rb" pod "a4c69512-a69f-4421-8372-2dbda197bcdf" (UID: "a4c69512-a69f-4421-8372-2dbda197bcdf"). InnerVolumeSpecName "kube-api-access-4r5rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:55:29.133588 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.133553 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-metrics-certs" pod "a4c69512-a69f-4421-8372-2dbda197bcdf" (UID: "a4c69512-a69f-4421-8372-2dbda197bcdf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:29.231406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.230472 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76221c7-dd2e-4ac4-9991-5b02a58f222c-kube-api-access-xblzh\" (UniqueName: \"kubernetes.io/projected/d76221c7-dd2e-4ac4-9991-5b02a58f222c-kube-api-access-xblzh\") pod \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\" (UID: \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\") " Nov 03 10:55:29.231406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.230865 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76221c7-dd2e-4ac4-9991-5b02a58f222c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76221c7-dd2e-4ac4-9991-5b02a58f222c-plugins-config-volume\") pod \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\" (UID: \"d76221c7-dd2e-4ac4-9991-5b02a58f222c\") " Nov 03 10:55:29.231658 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.231438 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:29.231658 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.231461 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c69512-a69f-4421-8372-2dbda197bcdf-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:29.231658 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.231476 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5rb\" (UniqueName: \"kubernetes.io/projected/a4c69512-a69f-4421-8372-2dbda197bcdf-kube-api-access-4r5rb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:29.233597 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.233522 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76221c7-dd2e-4ac4-9991-5b02a58f222c-plugins-config-volume" pod "d76221c7-dd2e-4ac4-9991-5b02a58f222c" (UID: "d76221c7-dd2e-4ac4-9991-5b02a58f222c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:55:29.235666 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.235643 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76221c7-dd2e-4ac4-9991-5b02a58f222c-kube-api-access-xblzh" pod "d76221c7-dd2e-4ac4-9991-5b02a58f222c" (UID: "d76221c7-dd2e-4ac4-9991-5b02a58f222c"). InnerVolumeSpecName "kube-api-access-xblzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:55:29.333251 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.333216 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76221c7-dd2e-4ac4-9991-5b02a58f222c-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:29.333251 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.333252 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xblzh\" (UniqueName: \"kubernetes.io/projected/d76221c7-dd2e-4ac4-9991-5b02a58f222c-kube-api-access-xblzh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:29.745759 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.745704 5086 generic.go:356] "Generic (PLEG): container finished" podID="a4c69512-a69f-4421-8372-2dbda197bcdf" containerID="30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22" exitCode=0 Nov 03 10:55:29.746241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.745920 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" event={"ID":"a4c69512-a69f-4421-8372-2dbda197bcdf","Type":"ContainerDied","Data":"30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22"} Nov 03 10:55:29.746241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.746021 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" event={"ID":"a4c69512-a69f-4421-8372-2dbda197bcdf","Type":"ContainerDied","Data":"f95e91ab6e7e1bb02d2d96a47c0f5f500613929331baedf7b470ee522d206d25"} Nov 03 10:55:29.746241 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.746061 5086 scope.go:122] "RemoveContainer" containerID="30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22" Nov 03 10:55:29.746455 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.746269 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl" Nov 03 10:55:29.751039 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.750885 5086 generic.go:356] "Generic (PLEG): container finished" podID="d76221c7-dd2e-4ac4-9991-5b02a58f222c" containerID="839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441" exitCode=0 Nov 03 10:55:29.751039 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.750978 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" event={"ID":"d76221c7-dd2e-4ac4-9991-5b02a58f222c","Type":"ContainerDied","Data":"839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441"} Nov 03 10:55:29.751039 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.751036 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" event={"ID":"d76221c7-dd2e-4ac4-9991-5b02a58f222c","Type":"ContainerDied","Data":"6cb21594430c3f66163f76fd9ffb28b7cc5ba973c96723f0a79c18ac26836b6d"} Nov 03 10:55:29.751242 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.751139 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg" Nov 03 10:55:29.769930 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.769908 5086 scope.go:122] "RemoveContainer" containerID="30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22" Nov 03 10:55:29.770708 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:55:29.770686 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22\": container with ID starting with 30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22 not found: ID does not exist" containerID="30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22" Nov 03 10:55:29.770793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.770715 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22"} err="failed to get container status \"30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22\": rpc error: code = NotFound desc = could not find container \"30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22\": container with ID starting with 30c4befc923ee65a4fc940b3270f6455178209f5c1d81e1e93aa068d7534fd22 not found: ID does not exist" Nov 03 10:55:29.770793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.770734 5086 scope.go:122] "RemoveContainer" containerID="839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441" Nov 03 10:55:29.780890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.780866 5086 scope.go:122] "RemoveContainer" containerID="839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441" Nov 03 10:55:29.781274 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:55:29.781249 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441\": container with ID starting with 839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441 not found: ID does not exist" containerID="839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441" Nov 03 10:55:29.781419 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.781283 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441"} err="failed to get container status \"839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441\": rpc error: code = NotFound desc = could not find container \"839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441\": container with ID starting with 839418bc6c830b4dbd330f4a0a629ddcfe7d297d39d05dc532298853f561a441 not found: ID does not exist" Nov 03 10:55:29.788758 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.788733 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl"] Nov 03 10:55:29.792470 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.792436 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18264e-75c978c94b-5d4sl"] Nov 03 10:55:29.809421 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.809390 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg"] Nov 03 10:55:29.812519 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:29.812488 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-5554bd494b-mdvqg"] Nov 03 10:55:30.171512 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:30.171482 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c69512-a69f-4421-8372-2dbda197bcdf" path="/var/lib/kubelet/pods/a4c69512-a69f-4421-8372-2dbda197bcdf/volumes" Nov 03 10:55:30.171904 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:30.171890 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76221c7-dd2e-4ac4-9991-5b02a58f222c" path="/var/lib/kubelet/pods/d76221c7-dd2e-4ac4-9991-5b02a58f222c/volumes" Nov 03 10:55:31.690329 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:31.690255 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr"] Nov 03 10:55:31.691225 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:31.690560 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" podUID="de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" containerName="maas-api" containerID="cri-o://3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108" gracePeriod=30 Nov 03 10:55:31.810483 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:31.810438 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj"] Nov 03 10:55:31.811321 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:31.811247 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" podUID="f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" containerName="maas-api" containerID="cri-o://3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8" gracePeriod=30 Nov 03 10:55:32.104122 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.104081 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:55:32.199947 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.199885 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:55:32.202187 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.202163 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls\") pod \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " Nov 03 10:55:32.202411 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.202397 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-openshift-service-ca\") pod \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " Nov 03 10:55:32.202468 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.202424 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-metrics-certs\") pod \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " Nov 03 10:55:32.202468 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.202458 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-kube-api-access-8t6wm\" (UniqueName: \"kubernetes.io/projected/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-kube-api-access-8t6wm\") pod \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\" (UID: \"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4\") " Nov 03 10:55:32.203016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.202772 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-openshift-service-ca" pod "de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" (UID: "de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:55:32.203016 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.202987 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:32.204764 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.204736 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls" pod "de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" (UID: "de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:32.204764 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.204751 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-metrics-certs" pod "de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" (UID: "de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:32.206316 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.206274 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-kube-api-access-8t6wm" pod "de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" (UID: "de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4"). InnerVolumeSpecName "kube-api-access-8t6wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:55:32.304460 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.304424 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-metrics-certs\") pod \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " Nov 03 10:55:32.304676 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.304483 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-openshift-service-ca\") pod \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " Nov 03 10:55:32.304676 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.304608 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-maas-api-tls\") pod \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " Nov 03 10:55:32.304807 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.304742 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-kube-api-access-pg9sr\" (UniqueName: \"kubernetes.io/projected/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-kube-api-access-pg9sr\") pod \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\" (UID: \"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0\") " Nov 03 10:55:32.304877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.304843 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-openshift-service-ca" pod "f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" (UID: "f322ca68-ef9d-48b8-bdf8-4549ab4d77d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:55:32.305254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.305230 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:32.305254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.305255 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:32.305629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.305268 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6wm\" (UniqueName: \"kubernetes.io/projected/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-kube-api-access-8t6wm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:32.305629 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.305281 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:32.306888 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.306863 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-kube-api-access-pg9sr" pod "f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" (UID: "f322ca68-ef9d-48b8-bdf8-4549ab4d77d0"). InnerVolumeSpecName "kube-api-access-pg9sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:55:32.307000 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.306935 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-maas-api-tls" pod "f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" (UID: "f322ca68-ef9d-48b8-bdf8-4549ab4d77d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:32.307000 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.306969 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-metrics-certs" pod "f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" (UID: "f322ca68-ef9d-48b8-bdf8-4549ab4d77d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:32.406653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.406617 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:32.406653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.406646 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:32.406653 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.406656 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg9sr\" (UniqueName: \"kubernetes.io/projected/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0-kube-api-access-pg9sr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:32.775194 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.775159 5086 generic.go:356] "Generic (PLEG): container finished" podID="de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" containerID="3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108" exitCode=0 Nov 03 10:55:32.775710 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.775209 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" Nov 03 10:55:32.775710 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.775245 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" event={"ID":"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4","Type":"ContainerDied","Data":"3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108"} Nov 03 10:55:32.775710 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.775313 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr" event={"ID":"de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4","Type":"ContainerDied","Data":"848484513bca93beffb54732a862d0174edd3629bb0318e64eb487470fe602d0"} Nov 03 10:55:32.775710 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.775330 5086 scope.go:122] "RemoveContainer" containerID="3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108" Nov 03 10:55:32.777074 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.777051 5086 generic.go:356] "Generic (PLEG): container finished" podID="f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" containerID="3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8" exitCode=0 Nov 03 10:55:32.777200 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.777094 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" Nov 03 10:55:32.777200 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.777127 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" event={"ID":"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0","Type":"ContainerDied","Data":"3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8"} Nov 03 10:55:32.777200 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.777167 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj" event={"ID":"f322ca68-ef9d-48b8-bdf8-4549ab4d77d0","Type":"ContainerDied","Data":"5037d6d7132637d515d00dc9c8e1b613e8c13a80809eb447bfd6cc39898a05ca"} Nov 03 10:55:32.787701 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.787680 5086 scope.go:122] "RemoveContainer" containerID="3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108" Nov 03 10:55:32.788022 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:55:32.787998 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108\": container with ID starting with 3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108 not found: ID does not exist" containerID="3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108" Nov 03 10:55:32.788108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.788035 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108"} err="failed to get container status \"3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108\": rpc error: code = NotFound desc = could not find container \"3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108\": container with ID starting with 3ace9007d292e17a2ecf244cd0cd88dcd6c0f59778d1a2f2e3481885d4384108 not found: ID does not exist" Nov 03 10:55:32.788108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.788056 5086 scope.go:122] "RemoveContainer" containerID="3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8" Nov 03 10:55:32.806095 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.806064 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr"] Nov 03 10:55:32.816757 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.816723 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-343e30-764589d66f-6zpmr"] Nov 03 10:55:32.820232 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.820190 5086 scope.go:122] "RemoveContainer" containerID="3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8" Nov 03 10:55:32.821382 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:55:32.821350 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8\": container with ID starting with 3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8 not found: ID does not exist" containerID="3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8" Nov 03 10:55:32.821561 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.821401 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8"} err="failed to get container status \"3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8\": rpc error: code = NotFound desc = could not find container \"3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8\": container with ID starting with 3cf04cb7b92e8cdbba43e9785250d75a2d37db888afb562873fd96322f344ad8 not found: ID does not exist" Nov 03 10:55:32.830628 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.830599 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj"] Nov 03 10:55:32.838091 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:32.838045 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cd6d01-6ff588678d-pmwlj"] Nov 03 10:55:34.172840 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:34.172808 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" path="/var/lib/kubelet/pods/de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4/volumes" Nov 03 10:55:34.173351 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:34.173335 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" path="/var/lib/kubelet/pods/f322ca68-ef9d-48b8-bdf8-4549ab4d77d0/volumes" Nov 03 10:55:48.591854 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:48.591801 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn"] Nov 03 10:55:49.583258 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:49.583220 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj"] Nov 03 10:55:49.583857 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:49.583493 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" podUID="4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" containerName="maas-api" containerID="cri-o://20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea" gracePeriod=30 Nov 03 10:55:49.834058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:49.833973 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6"] Nov 03 10:55:49.834538 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:49.834246 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" podUID="ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95" containerName="payload-processing" containerID="cri-o://1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1" gracePeriod=30 Nov 03 10:55:49.854558 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:49.854444 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6"] Nov 03 10:55:49.854737 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:49.854683 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" podUID="3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1" containerName="payload-pre-processing" containerID="cri-o://db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba" gracePeriod=30 Nov 03 10:55:50.023270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.023238 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:55:50.120206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.120121 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-kube-api-access-pjhbs\" (UniqueName: \"kubernetes.io/projected/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-kube-api-access-pjhbs\") pod \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " Nov 03 10:55:50.120465 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.120236 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-maas-api-tls\") pod \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " Nov 03 10:55:50.120465 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.120274 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-openshift-service-ca\") pod \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " Nov 03 10:55:50.120854 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.120674 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-metrics-certs\") pod \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\" (UID: \"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795\") " Nov 03 10:55:50.121239 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.121162 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-openshift-service-ca" pod "4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" (UID: "4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:55:50.124765 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.123686 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-maas-api-tls" pod "4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" (UID: "4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:50.125905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.125880 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-metrics-certs" pod "4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" (UID: "4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:55:50.133342 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.132383 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-kube-api-access-pjhbs" pod "4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" (UID: "4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795"). InnerVolumeSpecName "kube-api-access-pjhbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:55:50.152372 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.152341 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:55:50.223870 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.223828 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-kube-api-access-rtgf4\" (UniqueName: \"kubernetes.io/projected/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-kube-api-access-rtgf4\") pod \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\" (UID: \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\") " Nov 03 10:55:50.224044 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.223930 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-plugins-config-volume\") pod \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\" (UID: \"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95\") " Nov 03 10:55:50.224574 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.224553 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhbs\" (UniqueName: \"kubernetes.io/projected/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-kube-api-access-pjhbs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:50.224663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.224580 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:50.224663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.224595 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:50.224663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.224610 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:50.225068 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.225048 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-plugins-config-volume" pod "ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95" (UID: "ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:55:50.226998 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.226971 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-kube-api-access-rtgf4" pod "ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95" (UID: "ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95"). InnerVolumeSpecName "kube-api-access-rtgf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:55:50.250699 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.250673 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:55:50.344355 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.340708 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-kube-api-access-hh59b\" (UniqueName: \"kubernetes.io/projected/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-kube-api-access-hh59b\") pod \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\" (UID: \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\") " Nov 03 10:55:50.344355 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.340848 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-plugins-config-volume\") pod \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\" (UID: \"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1\") " Nov 03 10:55:50.344355 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.341449 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtgf4\" (UniqueName: \"kubernetes.io/projected/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-kube-api-access-rtgf4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:50.344355 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.341471 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:50.344355 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.341856 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-plugins-config-volume" pod "3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1" (UID: "3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:55:50.365728 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.365686 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-kube-api-access-hh59b" pod "3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1" (UID: "3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1"). InnerVolumeSpecName "kube-api-access-hh59b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:55:50.442722 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.442642 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh59b\" (UniqueName: \"kubernetes.io/projected/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-kube-api-access-hh59b\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:50.442722 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.442672 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:55:50.564635 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.564599 5086 generic.go:356] "Generic (PLEG): container finished" podID="4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" containerID="20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea" exitCode=0 Nov 03 10:55:50.564826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.564649 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" Nov 03 10:55:50.564826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.564687 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" event={"ID":"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795","Type":"ContainerDied","Data":"20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea"} Nov 03 10:55:50.564826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.564739 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj" event={"ID":"4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795","Type":"ContainerDied","Data":"3b00583fad095a8da9f46141d3568cd12ca74affabcdd60e1d106765b2b2a46f"} Nov 03 10:55:50.564826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.564760 5086 scope.go:122] "RemoveContainer" containerID="20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea" Nov 03 10:55:50.566283 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.566250 5086 generic.go:356] "Generic (PLEG): container finished" podID="3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1" containerID="db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba" exitCode=0 Nov 03 10:55:50.566429 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.566330 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" event={"ID":"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1","Type":"ContainerDied","Data":"db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba"} Nov 03 10:55:50.566429 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.566374 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" event={"ID":"3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1","Type":"ContainerDied","Data":"a15c9a228caf8cdac8510bc802a1220173ed798c946a4f899cf2b8c3ffe3b2c7"} Nov 03 10:55:50.566429 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.566334 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6" Nov 03 10:55:50.568256 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.568234 5086 generic.go:356] "Generic (PLEG): container finished" podID="ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95" containerID="1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1" exitCode=0 Nov 03 10:55:50.568418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.568305 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" Nov 03 10:55:50.568418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.568340 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" event={"ID":"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95","Type":"ContainerDied","Data":"1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1"} Nov 03 10:55:50.568418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.568380 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6" event={"ID":"ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95","Type":"ContainerDied","Data":"3347ea9f7ed223a5160fd2cd3f849d69df577cf2303d332bf64597523ed0412f"} Nov 03 10:55:50.578020 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.577999 5086 scope.go:122] "RemoveContainer" containerID="20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea" Nov 03 10:55:50.578322 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:55:50.578277 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea\": container with ID starting with 20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea not found: ID does not exist" containerID="20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea" Nov 03 10:55:50.578393 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.578319 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea"} err="failed to get container status \"20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea\": rpc error: code = NotFound desc = could not find container \"20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea\": container with ID starting with 20d1b7226eb7a718cbbdb989567c807af71a99ea3029e83c67931c02252e8fea not found: ID does not exist" Nov 03 10:55:50.578393 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.578336 5086 scope.go:122] "RemoveContainer" containerID="db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba" Nov 03 10:55:50.586718 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.586687 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj"] Nov 03 10:55:50.589674 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.589425 5086 scope.go:122] "RemoveContainer" containerID="db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba" Nov 03 10:55:50.589749 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:55:50.589725 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba\": container with ID starting with db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba not found: ID does not exist" containerID="db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba" Nov 03 10:55:50.589810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.589749 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba"} err="failed to get container status \"db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba\": rpc error: code = NotFound desc = could not find container \"db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba\": container with ID starting with db4ac6684e00fcdf68039eb9f1ca0226fd0ae6c9c00e22fb0d55ed3a0ab2e5ba not found: ID does not exist" Nov 03 10:55:50.589810 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.589775 5086 scope.go:122] "RemoveContainer" containerID="1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1" Nov 03 10:55:50.590550 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.590527 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8bbb6e-74d454f84d-626qj"] Nov 03 10:55:50.599577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.599557 5086 scope.go:122] "RemoveContainer" containerID="1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1" Nov 03 10:55:50.599885 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:55:50.599856 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1\": container with ID starting with 1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1 not found: ID does not exist" containerID="1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1" Nov 03 10:55:50.599954 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.599893 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1"} err="failed to get container status \"1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1\": rpc error: code = NotFound desc = could not find container \"1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1\": container with ID starting with 1fcee0225a169d147b63e42b740b001cd5cc94594eff72665b508579f41730c1 not found: ID does not exist" Nov 03 10:55:50.606559 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.606438 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6"] Nov 03 10:55:50.609821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.609797 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8bbb6e-779859794-72jv6"] Nov 03 10:55:50.619682 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.619658 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6"] Nov 03 10:55:50.624108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:50.624075 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-5fb9f876f5-gj6f6"] Nov 03 10:55:52.171958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:52.171927 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1" path="/var/lib/kubelet/pods/3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1/volumes" Nov 03 10:55:52.172404 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:52.172389 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" path="/var/lib/kubelet/pods/4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795/volumes" Nov 03 10:55:52.172792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:55:52.172778 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95" path="/var/lib/kubelet/pods/ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95/volumes" Nov 03 10:55:54.275196 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:55:54.275158 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:56:03.707330 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:03.707231 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" podUID="9d332a74-4865-489a-8fca-4dfa1ac23970" containerName="main" containerID="cri-o://0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2" gracePeriod=45 Nov 03 10:56:04.043988 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.043960 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:56:04.143209 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.143177 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-dshm\") pod \"9d332a74-4865-489a-8fca-4dfa1ac23970\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " Nov 03 10:56:04.143476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.143231 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-home\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-home\") pod \"9d332a74-4865-489a-8fca-4dfa1ac23970\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " Nov 03 10:56:04.143476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.143339 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d332a74-4865-489a-8fca-4dfa1ac23970-kube-api-access-mck68\" (UniqueName: \"kubernetes.io/projected/9d332a74-4865-489a-8fca-4dfa1ac23970-kube-api-access-mck68\") pod \"9d332a74-4865-489a-8fca-4dfa1ac23970\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " Nov 03 10:56:04.143476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.143377 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-model-cache\") pod \"9d332a74-4865-489a-8fca-4dfa1ac23970\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " Nov 03 10:56:04.143476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.143438 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-tmp-dir\") pod \"9d332a74-4865-489a-8fca-4dfa1ac23970\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " Nov 03 10:56:04.143797 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.143612 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d332a74-4865-489a-8fca-4dfa1ac23970-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d332a74-4865-489a-8fca-4dfa1ac23970-tls-certs\") pod \"9d332a74-4865-489a-8fca-4dfa1ac23970\" (UID: \"9d332a74-4865-489a-8fca-4dfa1ac23970\") " Nov 03 10:56:04.144753 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.144695 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-tmp-dir" pod "9d332a74-4865-489a-8fca-4dfa1ac23970" (UID: "9d332a74-4865-489a-8fca-4dfa1ac23970"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:56:04.145101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.145055 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-model-cache" pod "9d332a74-4865-489a-8fca-4dfa1ac23970" (UID: "9d332a74-4865-489a-8fca-4dfa1ac23970"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:56:04.145437 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.145408 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-home" pod "9d332a74-4865-489a-8fca-4dfa1ac23970" (UID: "9d332a74-4865-489a-8fca-4dfa1ac23970"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:56:04.146925 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.146900 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d332a74-4865-489a-8fca-4dfa1ac23970-tls-certs" pod "9d332a74-4865-489a-8fca-4dfa1ac23970" (UID: "9d332a74-4865-489a-8fca-4dfa1ac23970"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:56:04.147192 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.147145 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d332a74-4865-489a-8fca-4dfa1ac23970-kube-api-access-mck68" pod "9d332a74-4865-489a-8fca-4dfa1ac23970" (UID: "9d332a74-4865-489a-8fca-4dfa1ac23970"). InnerVolumeSpecName "kube-api-access-mck68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:56:04.147701 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.147678 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-dshm" pod "9d332a74-4865-489a-8fca-4dfa1ac23970" (UID: "9d332a74-4865-489a-8fca-4dfa1ac23970"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:56:04.264129 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.264027 5086 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d332a74-4865-489a-8fca-4dfa1ac23970-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:04.264894 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.264874 5086 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:04.265059 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.265047 5086 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:04.265548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.265529 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mck68\" (UniqueName: \"kubernetes.io/projected/9d332a74-4865-489a-8fca-4dfa1ac23970-kube-api-access-mck68\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:04.265711 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.265693 5086 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:04.265809 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.265798 5086 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d332a74-4865-489a-8fca-4dfa1ac23970-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:04.676132 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.676097 5086 generic.go:356] "Generic (PLEG): container finished" podID="9d332a74-4865-489a-8fca-4dfa1ac23970" containerID="0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2" exitCode=0 Nov 03 10:56:04.676361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.676145 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" Nov 03 10:56:04.676361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.676167 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" event={"ID":"9d332a74-4865-489a-8fca-4dfa1ac23970","Type":"ContainerDied","Data":"0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2"} Nov 03 10:56:04.676361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.676209 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn" event={"ID":"9d332a74-4865-489a-8fca-4dfa1ac23970","Type":"ContainerDied","Data":"cb1b0256cd5c41e17f8fc8e844a7a40745b3ff04d284e7f5c44ab25db5bf4124"} Nov 03 10:56:04.676361 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.676226 5086 scope.go:122] "RemoveContainer" containerID="0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2" Nov 03 10:56:04.686336 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.686319 5086 scope.go:122] "RemoveContainer" containerID="0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2" Nov 03 10:56:04.686659 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:56:04.686605 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2\": container with ID starting with 0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2 not found: ID does not exist" containerID="0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2" Nov 03 10:56:04.686760 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.686671 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2"} err="failed to get container status \"0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2\": rpc error: code = NotFound desc = could not find container \"0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2\": container with ID starting with 0a494b113e42b1ebb92d7485403b22fee27b24a67cf012db70a4a7b83c4f16b2 not found: ID does not exist" Nov 03 10:56:04.695545 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.695506 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn"] Nov 03 10:56:04.702846 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:04.702816 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-cd6d01/praxis-route-cd6d01-kserve-855fbcdfc7-2k2cn"] Nov 03 10:56:06.174061 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:06.174023 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d332a74-4865-489a-8fca-4dfa1ac23970" path="/var/lib/kubelet/pods/9d332a74-4865-489a-8fca-4dfa1ac23970/volumes" Nov 03 10:56:07.968191 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.968153 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c"] Nov 03 10:56:07.969149 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969124 5086 cpu_manager.go:414] "containerMap: removing container" podUID="bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" containerName="maas-api" Nov 03 10:56:07.969149 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969150 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" containerName="maas-api" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969171 5086 cpu_manager.go:414] "containerMap: removing container" podUID="9d332a74-4865-489a-8fca-4dfa1ac23970" containerName="main" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969179 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d332a74-4865-489a-8fca-4dfa1ac23970" containerName="main" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969190 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95" containerName="payload-processing" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969199 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95" containerName="payload-processing" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969221 5086 cpu_manager.go:414] "containerMap: removing container" podUID="a4c69512-a69f-4421-8372-2dbda197bcdf" containerName="maas-api" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969228 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c69512-a69f-4421-8372-2dbda197bcdf" containerName="maas-api" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969249 5086 cpu_manager.go:414] "containerMap: removing container" podUID="f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" containerName="maas-api" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969255 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" containerName="maas-api" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969284 5086 cpu_manager.go:414] "containerMap: removing container" podUID="3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1" containerName="payload-pre-processing" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969306 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1" containerName="payload-pre-processing" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969319 5086 cpu_manager.go:414] "containerMap: removing container" podUID="4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" containerName="maas-api" Nov 03 10:56:07.969332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969328 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" containerName="maas-api" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969361 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d76221c7-dd2e-4ac4-9991-5b02a58f222c" containerName="payload-processing" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969368 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76221c7-dd2e-4ac4-9991-5b02a58f222c" containerName="payload-processing" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969379 5086 cpu_manager.go:414] "containerMap: removing container" podUID="de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" containerName="maas-api" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969385 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" containerName="maas-api" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969528 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76221c7-dd2e-4ac4-9991-5b02a58f222c" containerName="payload-processing" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969542 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2f0e5f-a3f0-4fb9-a8b4-9427ced52bf4" containerName="maas-api" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969555 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3cfac0-27ec-4c71-8593-8ff3201fdc8a" containerName="maas-api" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969568 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d332a74-4865-489a-8fca-4dfa1ac23970" containerName="main" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969579 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebaff81c-2bc5-4bd6-bd9a-3179f11e1a95" containerName="payload-processing" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969591 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c69512-a69f-4421-8372-2dbda197bcdf" containerName="maas-api" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969604 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="f322ca68-ef9d-48b8-bdf8-4549ab4d77d0" containerName="maas-api" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969618 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="3741a6c0-d3e8-4f36-aed0-c4f3a5fb62e1" containerName="payload-pre-processing" Nov 03 10:56:07.969890 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.969626 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e622f9f-0bb3-4e1f-b1b8-f4c05d81e795" containerName="maas-api" Nov 03 10:56:07.993875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.993843 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:07.998369 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:07.998335 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c"] Nov 03 10:56:08.016536 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.016215 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-e92a08-serving-cert\"" Nov 03 10:56:08.042766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.042717 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4blg\" (UniqueName: \"kubernetes.io/projected/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-kube-api-access-s4blg\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.042930 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.042783 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.042930 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.042916 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.043069 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.042952 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.144577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.144536 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4blg\" (UniqueName: \"kubernetes.io/projected/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-kube-api-access-s4blg\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.144793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.144603 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.144793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.144731 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.144793 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.144766 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.145002 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:56:08.144979 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-serving-cert: secret "maas-api-e2e-shared-a-w4-e92a08-serving-cert" not found Nov 03 10:56:08.145081 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:56:08.145070 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls podName:9fcdd3bd-363b-4747-81c0-8bad0aee4a72 nodeName:}" failed. No retries permitted until 2026-11-03 10:56:08.645046197 +0000 UTC m=+1646.629917474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls") pod "maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" (UID: "9fcdd3bd-363b-4747-81c0-8bad0aee4a72") : secret "maas-api-e2e-shared-a-w4-e92a08-serving-cert" not found Nov 03 10:56:08.145654 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.145629 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.147623 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.147598 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.153217 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.153191 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4blg\" (UniqueName: \"kubernetes.io/projected/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-kube-api-access-s4blg\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.653281 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.653236 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.682318 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.675884 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:08.932966 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:08.932884 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:09.270103 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:09.270072 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c"] Nov 03 10:56:09.272786 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:56:09.272456 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcdd3bd_363b_4747_81c0_8bad0aee4a72.slice/crio-f474f60eafdc853d033cb9fe43cbf036d399ee1ad3badfff3a209e70e5e9e94d WatchSource:0}: Error finding container f474f60eafdc853d033cb9fe43cbf036d399ee1ad3badfff3a209e70e5e9e94d: Status 404 returned error can't find the container with id f474f60eafdc853d033cb9fe43cbf036d399ee1ad3badfff3a209e70e5e9e94d Nov 03 10:56:09.280844 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:09.280791 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:56:09.734213 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:09.734172 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" event={"ID":"9fcdd3bd-363b-4747-81c0-8bad0aee4a72","Type":"ContainerStarted","Data":"f474f60eafdc853d033cb9fe43cbf036d399ee1ad3badfff3a209e70e5e9e94d"} Nov 03 10:56:10.741600 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:10.741558 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" event={"ID":"9fcdd3bd-363b-4747-81c0-8bad0aee4a72","Type":"ContainerStarted","Data":"39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e"} Nov 03 10:56:10.742254 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:10.741642 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:10.759850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:10.759798 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" podStartSLOduration=3.2507099950000002 podStartE2EDuration="3.75978214s" totalImagesPullingTime="509.072145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:10.757618435 +0000 UTC m=+1648.742489720" watchObservedRunningTime="2026-11-03 10:56:10.75978214 +0000 UTC m=+1648.744653482" Nov 03 10:56:13.868745 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:13.868705 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b"] Nov 03 10:56:13.873473 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:13.873444 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:13.882927 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:13.882888 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b"] Nov 03 10:56:13.883143 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:13.883058 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-6012b5-dockercfg-s2jl4\"" Nov 03 10:56:13.937137 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:13.937096 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-6012b5\"" Nov 03 10:56:13.961214 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:13.961163 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxcz\" (UniqueName: \"kubernetes.io/projected/c1228536-409d-41c3-aab5-b70ea3c1f4e6-kube-api-access-zqxcz\") pod \"payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b\" (UID: \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:13.961455 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:13.961274 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1228536-409d-41c3-aab5-b70ea3c1f4e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b\" (UID: \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:14.062404 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.062369 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxcz\" (UniqueName: \"kubernetes.io/projected/c1228536-409d-41c3-aab5-b70ea3c1f4e6-kube-api-access-zqxcz\") pod \"payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b\" (UID: \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:14.062583 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.062452 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1228536-409d-41c3-aab5-b70ea3c1f4e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b\" (UID: \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:14.063276 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.063252 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1228536-409d-41c3-aab5-b70ea3c1f4e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b\" (UID: \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:14.070663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.070641 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxcz\" (UniqueName: \"kubernetes.io/projected/c1228536-409d-41c3-aab5-b70ea3c1f4e6-kube-api-access-zqxcz\") pod \"payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b\" (UID: \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:14.197515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.197419 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:14.505452 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.505417 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b"] Nov 03 10:56:14.506831 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:56:14.506801 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1228536_409d_41c3_aab5_b70ea3c1f4e6.slice/crio-ae659cf9b3707afaf249a472edff8339caa7dff0657b3aeaf7f9e198e4e0cdf2 WatchSource:0}: Error finding container ae659cf9b3707afaf249a472edff8339caa7dff0657b3aeaf7f9e198e4e0cdf2: Status 404 returned error can't find the container with id ae659cf9b3707afaf249a472edff8339caa7dff0657b3aeaf7f9e198e4e0cdf2 Nov 03 10:56:14.801328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.801230 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" event={"ID":"c1228536-409d-41c3-aab5-b70ea3c1f4e6","Type":"ContainerStarted","Data":"48b0f73f069fc392f03a44cb89ff41b45a99a274c407882c38df58ef849dfe82"} Nov 03 10:56:14.801328 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.801284 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" event={"ID":"c1228536-409d-41c3-aab5-b70ea3c1f4e6","Type":"ContainerStarted","Data":"ae659cf9b3707afaf249a472edff8339caa7dff0657b3aeaf7f9e198e4e0cdf2"} Nov 03 10:56:14.801519 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.801462 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:14.818823 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:14.818772 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" podStartSLOduration=1.818759142 podStartE2EDuration="1.818759142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:14.815487755 +0000 UTC m=+1652.800359038" watchObservedRunningTime="2026-11-03 10:56:14.818759142 +0000 UTC m=+1652.803630425" Nov 03 10:56:16.755574 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:16.755545 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 10:56:17.646537 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.646502 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47"] Nov 03 10:56:17.651140 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.651118 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.658728 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.658693 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-6012b5-serving-cert\"" Nov 03 10:56:17.663510 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.663485 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47"] Nov 03 10:56:17.724346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.724289 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.724526 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.724417 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdng\" (UniqueName: \"kubernetes.io/projected/fd727856-e40a-4dc5-9c67-f44e8f89e123-kube-api-access-fbdng\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.724526 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.724496 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.724679 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.724652 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd727856-e40a-4dc5-9c67-f44e8f89e123-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.825679 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.825646 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.826175 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.825789 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd727856-e40a-4dc5-9c67-f44e8f89e123-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.826175 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.825839 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.826175 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.825913 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdng\" (UniqueName: \"kubernetes.io/projected/fd727856-e40a-4dc5-9c67-f44e8f89e123-kube-api-access-fbdng\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.826175 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:56:17.826012 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-serving-cert: secret "maas-api-e2e-shared-a-w1-6012b5-serving-cert" not found Nov 03 10:56:17.826175 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:56:17.826090 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls podName:fd727856-e40a-4dc5-9c67-f44e8f89e123 nodeName:}" failed. No retries permitted until 2026-11-03 10:56:18.326069725 +0000 UTC m=+1656.310941002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls") pod "maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" (UID: "fd727856-e40a-4dc5-9c67-f44e8f89e123") : secret "maas-api-e2e-shared-a-w1-6012b5-serving-cert" not found Nov 03 10:56:17.826530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.826510 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd727856-e40a-4dc5-9c67-f44e8f89e123-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.828520 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.828493 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:17.840571 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:17.840535 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdng\" (UniqueName: \"kubernetes.io/projected/fd727856-e40a-4dc5-9c67-f44e8f89e123-kube-api-access-fbdng\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:18.334616 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:18.334052 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:18.338607 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:18.338570 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:18.575826 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:18.575792 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:18.857177 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:18.857149 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47"] Nov 03 10:56:18.857679 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:56:18.857650 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd727856_e40a_4dc5_9c67_f44e8f89e123.slice/crio-4870334d3a3f8b5cb628cb375e79c24b6b4b39b52b823759565cf821fee186db WatchSource:0}: Error finding container 4870334d3a3f8b5cb628cb375e79c24b6b4b39b52b823759565cf821fee186db: Status 404 returned error can't find the container with id 4870334d3a3f8b5cb628cb375e79c24b6b4b39b52b823759565cf821fee186db Nov 03 10:56:19.839718 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:19.839676 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" event={"ID":"fd727856-e40a-4dc5-9c67-f44e8f89e123","Type":"ContainerStarted","Data":"9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6"} Nov 03 10:56:19.839899 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:19.839776 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:19.839899 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:19.839798 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" event={"ID":"fd727856-e40a-4dc5-9c67-f44e8f89e123","Type":"ContainerStarted","Data":"4870334d3a3f8b5cb628cb375e79c24b6b4b39b52b823759565cf821fee186db"} Nov 03 10:56:19.855336 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:19.855251 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" podStartSLOduration=2.396091724 podStartE2EDuration="2.855234448s" totalImagesPullingTime="459.142724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:19.854224074 +0000 UTC m=+1657.839095356" watchObservedRunningTime="2026-11-03 10:56:19.855234448 +0000 UTC m=+1657.840105736" Nov 03 10:56:25.108123 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:56:25.104510 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:56:25.462375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:25.452613 5086 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:56:25.462375 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:25.452696 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:56:25.813013 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:25.812869 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 10:56:25.879982 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:25.879944 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 10:56:38.830146 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:38.830111 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8"] Nov 03 10:56:38.834923 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:38.834903 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:38.842269 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:38.842244 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-fcc2ce-serving-cert\"" Nov 03 10:56:38.844670 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:38.844642 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8"] Nov 03 10:56:38.959986 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:38.959952 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:38.960174 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:38.959995 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj88b\" (UniqueName: \"kubernetes.io/projected/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-kube-api-access-mj88b\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:38.960174 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:38.960108 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:38.960259 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:38.960201 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.061105 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.061066 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.061105 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.061110 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj88b\" (UniqueName: \"kubernetes.io/projected/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-kube-api-access-mj88b\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.061441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.061140 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.061441 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.061178 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.061928 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.061900 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.063935 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.063907 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.064117 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.064095 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.068315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.068278 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj88b\" (UniqueName: \"kubernetes.io/projected/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-kube-api-access-mj88b\") pod \"maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.150613 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.150530 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:39.439972 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:39.439488 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8"] Nov 03 10:56:40.185205 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:40.185115 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" event={"ID":"c9670a7d-3aa9-4225-9cbb-783d6dcc4734","Type":"ContainerStarted","Data":"72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc"} Nov 03 10:56:40.185205 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:40.185176 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:40.185205 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:40.185193 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" event={"ID":"c9670a7d-3aa9-4225-9cbb-783d6dcc4734","Type":"ContainerStarted","Data":"86d4fd6df83e137478f9aecc720c9d168591fabe9ae26c0522f1f28e0565381b"} Nov 03 10:56:40.203502 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:40.203439 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" podStartSLOduration=1.738125883 podStartE2EDuration="2.20341968s" totalImagesPullingTime="465.293797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:40.198978526 +0000 UTC m=+1678.183849808" watchObservedRunningTime="2026-11-03 10:56:40.20341968 +0000 UTC m=+1678.188290963" Nov 03 10:56:46.211998 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:46.211965 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 10:56:47.949080 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:47.949046 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx"] Nov 03 10:56:47.955956 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:47.955928 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:47.970563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:47.968472 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-cc8214-serving-cert\"" Nov 03 10:56:47.977900 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:47.977870 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx"] Nov 03 10:56:48.030128 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.030098 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.030324 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.030163 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.030324 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.030258 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76t5b\" (UniqueName: \"kubernetes.io/projected/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-kube-api-access-76t5b\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.030586 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.030570 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.132011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.131969 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.132011 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.132012 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76t5b\" (UniqueName: \"kubernetes.io/projected/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-kube-api-access-76t5b\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.132288 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.132197 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.132288 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.132248 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.132952 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.132913 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.161514 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.140040 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.161514 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.140218 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.161514 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.157083 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76t5b\" (UniqueName: \"kubernetes.io/projected/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-kube-api-access-76t5b\") pod \"maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.287273 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.287189 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:48.427205 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.427164 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp"] Nov 03 10:56:48.449515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.447737 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.450834 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.450800 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp"] Nov 03 10:56:48.553169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.553056 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.553169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.553146 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.553645 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.553175 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea01029-a736-4113-b371-eb957f0b566f-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.553645 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.553204 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.553645 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.553230 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsxn4\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-kube-api-access-jsxn4\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.553645 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.553417 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.553645 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.553452 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea01029-a736-4113-b371-eb957f0b566f-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.553645 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.553485 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.553645 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.553649 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.629471 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.629430 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx"] Nov 03 10:56:48.631066 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:56:48.631038 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70193f5_6efd_4009_a8a3_2d1405f4ce7c.slice/crio-397eab29292a6ea639d68c74fa283126497871399e41d5f63e08941d7f3e02ce WatchSource:0}: Error finding container 397eab29292a6ea639d68c74fa283126497871399e41d5f63e08941d7f3e02ce: Status 404 returned error can't find the container with id 397eab29292a6ea639d68c74fa283126497871399e41d5f63e08941d7f3e02ce Nov 03 10:56:48.655611 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.655585 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.655743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.655634 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.655743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.655667 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea01029-a736-4113-b371-eb957f0b566f-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.655743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.655692 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.655743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.655716 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsxn4\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-kube-api-access-jsxn4\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.655924 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.655829 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.655924 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.655858 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea01029-a736-4113-b371-eb957f0b566f-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.655924 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.655891 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.656062 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.656048 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.656378 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.656353 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.656492 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.656393 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.656492 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.656417 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.656492 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.656477 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.656752 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.656733 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea01029-a736-4113-b371-eb957f0b566f-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.658188 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.658165 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.658541 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.658523 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea01029-a736-4113-b371-eb957f0b566f-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.674078 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.674046 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.681509 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.681481 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsxn4\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-kube-api-access-jsxn4\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:48.771590 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:48.771549 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:49.121896 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.121870 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp"] Nov 03 10:56:49.132886 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.132324 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:56:49.132886 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.132424 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:56:49.132886 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.132471 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 10:56:49.379521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.379476 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" event={"ID":"2ea01029-a736-4113-b371-eb957f0b566f","Type":"ContainerStarted","Data":"754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7"} Nov 03 10:56:49.380008 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.379548 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" event={"ID":"2ea01029-a736-4113-b371-eb957f0b566f","Type":"ContainerStarted","Data":"fc80ad30012dabacf914c4cc6f246dc9e3c15b26417a6b849c9bede2347d3935"} Nov 03 10:56:49.384005 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.383954 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" event={"ID":"b70193f5-6efd-4009-a8a3-2d1405f4ce7c","Type":"ContainerStarted","Data":"41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550"} Nov 03 10:56:49.384171 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.384063 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" event={"ID":"b70193f5-6efd-4009-a8a3-2d1405f4ce7c","Type":"ContainerStarted","Data":"397eab29292a6ea639d68c74fa283126497871399e41d5f63e08941d7f3e02ce"} Nov 03 10:56:49.385053 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.385028 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:49.406431 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.406327 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" podStartSLOduration=1.4063132600000001 podStartE2EDuration="1.40631326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:49.401996506 +0000 UTC m=+1687.386868048" watchObservedRunningTime="2026-11-03 10:56:49.40631326 +0000 UTC m=+1687.391184534" Nov 03 10:56:49.419101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.419042 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" podStartSLOduration=2.005054261 podStartE2EDuration="2.419025324s" totalImagesPullingTime="413.971063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:49.417833623 +0000 UTC m=+1687.402704908" watchObservedRunningTime="2026-11-03 10:56:49.419025324 +0000 UTC m=+1687.403896608" Nov 03 10:56:49.773101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:49.772997 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:50.774413 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:50.774379 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 10:56:50.774803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:50.774434 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 10:56:51.776819 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:51.776785 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 10:56:51.777515 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:51.777480 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 10:56:52.772959 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:52.772923 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 10:56:52.773188 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:52.772981 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 10:56:53.772655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:53.772618 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 10:56:53.773153 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:53.772680 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 10:56:53.983242 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:53.983206 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms"] Nov 03 10:56:53.983541 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:53.983508 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" podUID="5304f031-410d-4251-8c2b-2391978c4884" containerName="maas-api" containerID="cri-o://dd0762fda1f194244c854e008a5283e70c1092c06c72f0f33275e0fab49f8618" gracePeriod=30 Nov 03 10:56:54.464327 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:56:54.462856 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:56:54.464487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.464385 5086 generic.go:356] "Generic (PLEG): container finished" podID="5304f031-410d-4251-8c2b-2391978c4884" containerID="dd0762fda1f194244c854e008a5283e70c1092c06c72f0f33275e0fab49f8618" exitCode=0 Nov 03 10:56:54.464487 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.464476 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" event={"ID":"5304f031-410d-4251-8c2b-2391978c4884","Type":"ContainerDied","Data":"dd0762fda1f194244c854e008a5283e70c1092c06c72f0f33275e0fab49f8618"} Nov 03 10:56:54.464959 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.464514 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" event={"ID":"5304f031-410d-4251-8c2b-2391978c4884","Type":"ContainerDied","Data":"027d927763d4deb1014b25da8143f47cd9aa981e0255a6928c0517b45786de58"} Nov 03 10:56:54.464959 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.464530 5086 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027d927763d4deb1014b25da8143f47cd9aa981e0255a6928c0517b45786de58" Nov 03 10:56:54.464959 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.464545 5086 scope.go:122] "RemoveContainer" containerID="447642f448751a4081c82acf82062a27cb045082a142114633369b392d1271d2" Nov 03 10:56:54.467474 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.467450 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:56:54.590827 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.590790 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5304f031-410d-4251-8c2b-2391978c4884-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5304f031-410d-4251-8c2b-2391978c4884-openshift-service-ca\") pod \"5304f031-410d-4251-8c2b-2391978c4884\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " Nov 03 10:56:54.590982 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.590922 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-maas-api-tls\") pod \"5304f031-410d-4251-8c2b-2391978c4884\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " Nov 03 10:56:54.591063 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.591047 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5304f031-410d-4251-8c2b-2391978c4884-kube-api-access-2cq5k\" (UniqueName: \"kubernetes.io/projected/5304f031-410d-4251-8c2b-2391978c4884-kube-api-access-2cq5k\") pod \"5304f031-410d-4251-8c2b-2391978c4884\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " Nov 03 10:56:54.591162 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.591148 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-metrics-certs\") pod \"5304f031-410d-4251-8c2b-2391978c4884\" (UID: \"5304f031-410d-4251-8c2b-2391978c4884\") " Nov 03 10:56:54.591237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.591217 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5304f031-410d-4251-8c2b-2391978c4884-openshift-service-ca" pod "5304f031-410d-4251-8c2b-2391978c4884" (UID: "5304f031-410d-4251-8c2b-2391978c4884"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:56:54.591679 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.591658 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5304f031-410d-4251-8c2b-2391978c4884-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:54.593139 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.593109 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-maas-api-tls" pod "5304f031-410d-4251-8c2b-2391978c4884" (UID: "5304f031-410d-4251-8c2b-2391978c4884"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:56:54.593250 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.593134 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5304f031-410d-4251-8c2b-2391978c4884-kube-api-access-2cq5k" pod "5304f031-410d-4251-8c2b-2391978c4884" (UID: "5304f031-410d-4251-8c2b-2391978c4884"). InnerVolumeSpecName "kube-api-access-2cq5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:56:54.593250 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.593173 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-metrics-certs" pod "5304f031-410d-4251-8c2b-2391978c4884" (UID: "5304f031-410d-4251-8c2b-2391978c4884"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:56:54.695683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.694071 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:54.695683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.694112 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5304f031-410d-4251-8c2b-2391978c4884-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:54.695683 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.694125 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cq5k\" (UniqueName: \"kubernetes.io/projected/5304f031-410d-4251-8c2b-2391978c4884-kube-api-access-2cq5k\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:56:54.772746 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.772711 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 10:56:54.773153 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:54.772769 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 10:56:55.474762 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:55.474733 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms" Nov 03 10:56:55.508602 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:55.508569 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms"] Nov 03 10:56:55.513979 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:55.513952 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a35a97-6754c57d6c-cwvms"] Nov 03 10:56:55.772740 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:55.772650 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 10:56:55.772740 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:55.772724 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 10:56:56.188364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.188288 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5304f031-410d-4251-8c2b-2391978c4884" path="/var/lib/kubelet/pods/5304f031-410d-4251-8c2b-2391978c4884/volumes" Nov 03 10:56:56.201966 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.201933 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx"] Nov 03 10:56:56.202724 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.202705 5086 cpu_manager.go:414] "containerMap: removing container" podUID="5304f031-410d-4251-8c2b-2391978c4884" containerName="maas-api" Nov 03 10:56:56.202811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.202725 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5304f031-410d-4251-8c2b-2391978c4884" containerName="maas-api" Nov 03 10:56:56.202811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.202739 5086 cpu_manager.go:414] "containerMap: removing container" podUID="5304f031-410d-4251-8c2b-2391978c4884" containerName="maas-api" Nov 03 10:56:56.202811 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.202745 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5304f031-410d-4251-8c2b-2391978c4884" containerName="maas-api" Nov 03 10:56:56.202966 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.202852 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="5304f031-410d-4251-8c2b-2391978c4884" containerName="maas-api" Nov 03 10:56:56.202966 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.202859 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="5304f031-410d-4251-8c2b-2391978c4884" containerName="maas-api" Nov 03 10:56:56.231313 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.230809 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx"] Nov 03 10:56:56.248359 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.233692 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.267501 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.267467 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-47a8a4-serving-cert\"" Nov 03 10:56:56.333481 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.333397 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxkw\" (UniqueName: \"kubernetes.io/projected/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-kube-api-access-vtxkw\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.333681 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.333543 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.333774 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.333719 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.333974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.333947 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.407550 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.407522 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 10:56:56.435233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.435196 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.435510 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.435436 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.436850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.436518 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.437564 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.437539 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxkw\" (UniqueName: \"kubernetes.io/projected/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-kube-api-access-vtxkw\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.437763 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.437744 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.448711 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.446506 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.448711 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.447071 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.498924 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.498886 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxkw\" (UniqueName: \"kubernetes.io/projected/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-kube-api-access-vtxkw\") pod \"maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.563553 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.563520 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:56.841718 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.841688 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:56.842197 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.842170 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:56.843008 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.842988 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 10:56:56.884554 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:56:56.884523 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb7b2a6_ee19_4f89_8fca_23ba1c8fb3c2.slice/crio-59d0a939662f32f4d30ba8b712706e0d9acadffc83c2f985d36b4b59518f17d4 WatchSource:0}: Error finding container 59d0a939662f32f4d30ba8b712706e0d9acadffc83c2f985d36b4b59518f17d4: Status 404 returned error can't find the container with id 59d0a939662f32f4d30ba8b712706e0d9acadffc83c2f985d36b4b59518f17d4 Nov 03 10:56:56.892713 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:56.892665 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx"] Nov 03 10:56:57.581215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:57.580496 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" event={"ID":"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2","Type":"ContainerStarted","Data":"cc891f7044dd0e78f582541d870f01af59163a9a1b434c86621f286495486c27"} Nov 03 10:56:57.581215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:57.580563 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:56:57.581215 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:57.580584 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" event={"ID":"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2","Type":"ContainerStarted","Data":"59d0a939662f32f4d30ba8b712706e0d9acadffc83c2f985d36b4b59518f17d4"} Nov 03 10:56:57.598690 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:57.598631 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" podStartSLOduration=1.159613423 podStartE2EDuration="1.598612348s" totalImagesPullingTime="438.998925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:57.595927729 +0000 UTC m=+1695.580799013" watchObservedRunningTime="2026-11-03 10:56:57.598612348 +0000 UTC m=+1695.583483632" Nov 03 10:56:59.330024 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.329984 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v"] Nov 03 10:56:59.335791 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.335767 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.344556 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.344528 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e92a08\"/\"openshift-service-ca.crt\"" Nov 03 10:56:59.344556 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.344528 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e92a08\"/\"kube-root-ca.crt\"" Nov 03 10:56:59.344766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.344597 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e92a08\"/\"default-dockercfg-k296c\"" Nov 03 10:56:59.344766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.344675 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e92a08\"/\"auth-test-model-e92a08-kserve-self-signed-certs\"" Nov 03 10:56:59.345764 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.345735 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v"] Nov 03 10:56:59.500507 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.500438 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tls-certs\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.500733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.500527 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-dshm\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.500733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.500612 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tmp-dir\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.500733 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.500650 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-model-cache\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.501101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.500777 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-home\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.501101 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.500894 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgs9m\" (UniqueName: \"kubernetes.io/projected/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-kube-api-access-kgs9m\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.602906 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.602799 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgs9m\" (UniqueName: \"kubernetes.io/projected/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-kube-api-access-kgs9m\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.602906 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.602901 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tls-certs\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.603165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.602964 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-dshm\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.603165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.603070 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tmp-dir\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.603165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.603109 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-model-cache\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.603710 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.603682 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tmp-dir\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.603837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.603707 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-model-cache\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.603837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.603817 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-home\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.604224 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.604198 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-home\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.605848 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.605828 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-dshm\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.606221 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.606202 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tls-certs\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.610703 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.610676 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgs9m\" (UniqueName: \"kubernetes.io/projected/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-kube-api-access-kgs9m\") pod \"auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.660920 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.657402 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:56:59.961423 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:56:59.961359 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v"] Nov 03 10:57:00.596322 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:00.596253 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" event={"ID":"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c","Type":"ContainerStarted","Data":"b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7"} Nov 03 10:57:00.596772 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:00.596335 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" event={"ID":"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c","Type":"ContainerStarted","Data":"08a448502946d33ea0d66565b2703045f49d7c8ece72a138043a674cd9cb4565"} Nov 03 10:57:00.614077 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:00.614022 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" podStartSLOduration=1.6140089359999998 podStartE2EDuration="1.614008936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:00.612011289 +0000 UTC m=+1698.596882566" watchObservedRunningTime="2026-11-03 10:57:00.614008936 +0000 UTC m=+1698.598880219" Nov 03 10:57:00.658379 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:00.658345 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:57:03.587869 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:03.587834 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 10:57:05.630401 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.630356 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r"] Nov 03 10:57:05.636136 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.636107 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.643169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.640542 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-ce5555-serving-cert\"" Nov 03 10:57:05.643169 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.642035 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r"] Nov 03 10:57:05.710958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.710921 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.711112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.711046 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.711112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.711093 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86mb\" (UniqueName: \"kubernetes.io/projected/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-kube-api-access-j86mb\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.711252 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.711234 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.812665 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.812626 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.812861 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.812707 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.812861 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.812736 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86mb\" (UniqueName: \"kubernetes.io/projected/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-kube-api-access-j86mb\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.812861 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.812806 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.813551 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.813524 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.815270 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.815247 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.815380 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.815329 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.820850 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.820820 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86mb\" (UniqueName: \"kubernetes.io/projected/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-kube-api-access-j86mb\") pod \"maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:05.961162 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:05.961068 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:06.251958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:06.251924 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r"] Nov 03 10:57:06.252526 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:57:06.252502 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fc1bd1_f950_46a7_8ec3_54555a42e1d0.slice/crio-0179954c429f76d17f687ba49a5f9f2fc7c7e7f2c3d7f98cbce8ffdd488dadbd WatchSource:0}: Error finding container 0179954c429f76d17f687ba49a5f9f2fc7c7e7f2c3d7f98cbce8ffdd488dadbd: Status 404 returned error can't find the container with id 0179954c429f76d17f687ba49a5f9f2fc7c7e7f2c3d7f98cbce8ffdd488dadbd Nov 03 10:57:06.682007 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:06.681970 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" event={"ID":"20fc1bd1-f950-46a7-8ec3-54555a42e1d0","Type":"ContainerStarted","Data":"0179954c429f76d17f687ba49a5f9f2fc7c7e7f2c3d7f98cbce8ffdd488dadbd"} Nov 03 10:57:07.692356 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:07.692311 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" event={"ID":"20fc1bd1-f950-46a7-8ec3-54555a42e1d0","Type":"ContainerStarted","Data":"2dccc69098e8ccf41d1a80bf143dc803c45685bcf9599509576e0c27e5caef73"} Nov 03 10:57:07.692778 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:07.692394 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:09.657663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:09.657624 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:57:09.670406 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:09.670381 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:57:09.686198 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:09.686149 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" podStartSLOduration=4.213691621 podStartE2EDuration="4.686134018s" totalImagesPullingTime="472.442397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:07.709901272 +0000 UTC m=+1705.694772556" watchObservedRunningTime="2026-11-03 10:57:09.686134018 +0000 UTC m=+1707.671005301" Nov 03 10:57:09.717315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:09.717269 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 10:57:13.706048 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:13.706008 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 10:57:20.256422 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.256391 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8"] Nov 03 10:57:20.262220 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.262195 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.271478 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.271443 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-fcc2ce\"/\"openshift-service-ca.crt\"" Nov 03 10:57:20.271649 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.271571 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-fcc2ce\"/\"kube-root-ca.crt\"" Nov 03 10:57:20.276765 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.276726 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8"] Nov 03 10:57:20.285418 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.285384 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-fcc2ce\"/\"default-dockercfg-wvbsw\"" Nov 03 10:57:20.286340 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.286307 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-fcc2ce\"/\"auth-test-model-fcc2ce-kserve-self-signed-certs\"" Nov 03 10:57:20.345077 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.345038 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-home\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.345260 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.345177 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1a91d-c562-47ab-8919-6db7cb914802-tls-certs\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.345260 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.345225 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7cnt\" (UniqueName: \"kubernetes.io/projected/66c1a91d-c562-47ab-8919-6db7cb914802-kube-api-access-v7cnt\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.345391 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.345376 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-dshm\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.345446 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.345434 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-model-cache\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.345587 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.345569 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-tmp-dir\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.447095 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.447056 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-dshm\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.447282 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.447125 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-model-cache\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.447282 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.447179 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-tmp-dir\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.447282 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.447226 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-home\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.447497 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.447356 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1a91d-c562-47ab-8919-6db7cb914802-tls-certs\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.447497 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.447396 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7cnt\" (UniqueName: \"kubernetes.io/projected/66c1a91d-c562-47ab-8919-6db7cb914802-kube-api-access-v7cnt\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.447639 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.447619 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-model-cache\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.447706 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.447682 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-tmp-dir\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.447777 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.447752 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-home\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.449526 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.449499 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-dshm\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.449792 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.449777 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1a91d-c562-47ab-8919-6db7cb914802-tls-certs\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.454801 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.454781 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7cnt\" (UniqueName: \"kubernetes.io/projected/66c1a91d-c562-47ab-8919-6db7cb914802-kube-api-access-v7cnt\") pod \"auth-test-model-fcc2ce-kserve-865d468cbf-pkml8\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.583982 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.583936 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:20.901201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:20.901162 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8"] Nov 03 10:57:21.841866 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:21.841825 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" event={"ID":"66c1a91d-c562-47ab-8919-6db7cb914802","Type":"ContainerStarted","Data":"d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55"} Nov 03 10:57:21.842468 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:21.842439 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" event={"ID":"66c1a91d-c562-47ab-8919-6db7cb914802","Type":"ContainerStarted","Data":"40826f71c166118e810ff13ac1390e20a7086dfaf20265ae89131f8d8a225ea1"} Nov 03 10:57:21.873910 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:21.873629 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" podStartSLOduration=1.873604581 podStartE2EDuration="1.873604581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:21.865203142 +0000 UTC m=+1719.850074429" watchObservedRunningTime="2026-11-03 10:57:21.873604581 +0000 UTC m=+1719.858475876" Nov 03 10:57:22.585398 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:22.585363 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:24.277453 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:57:24.277407 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:57:29.275884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.275847 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr"] Nov 03 10:57:29.297958 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.297887 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.298825 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.298797 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr"] Nov 03 10:57:29.302974 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.302944 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-e11de4-serving-cert\"" Nov 03 10:57:29.420607 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.420571 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.420824 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.420775 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.420824 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.420810 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrkd\" (UniqueName: \"kubernetes.io/projected/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-kube-api-access-qwrkd\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.420991 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.420975 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.521906 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.521868 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.522114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.522004 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.522114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.522026 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrkd\" (UniqueName: \"kubernetes.io/projected/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-kube-api-access-qwrkd\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.522233 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.522120 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.522891 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.522861 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.525696 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.525658 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.527142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.527050 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.530437 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.530410 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrkd\" (UniqueName: \"kubernetes.io/projected/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-kube-api-access-qwrkd\") pod \"maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.615532 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.615478 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:29.942594 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:29.942561 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr"] Nov 03 10:57:30.586183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:30.585096 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:30.603364 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:30.603325 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:30.921933 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:30.921242 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" event={"ID":"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6","Type":"ContainerStarted","Data":"7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b"} Nov 03 10:57:30.921933 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:30.921312 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" event={"ID":"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6","Type":"ContainerStarted","Data":"a280a2978a7456d8748486e5819728ad3d102f9d9a780bd261e51c10bcdac510"} Nov 03 10:57:30.921933 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:30.921361 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:30.935997 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:30.935965 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 10:57:30.970844 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:30.970786 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" podStartSLOduration=1.485900534 podStartE2EDuration="1.970770559s" totalImagesPullingTime="484.870025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:30.94391783 +0000 UTC m=+1728.928789114" watchObservedRunningTime="2026-11-03 10:57:30.970770559 +0000 UTC m=+1728.955641891" Nov 03 10:57:31.261381 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.260207 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk"] Nov 03 10:57:31.289315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.288571 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.310320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.298597 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-3ba121-serving-cert\"" Nov 03 10:57:31.355349 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.353017 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk"] Nov 03 10:57:31.427962 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.427605 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-openshift-service-ca\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.428600 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.428570 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-metrics-certs\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.429403 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.429201 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llk8\" (UniqueName: \"kubernetes.io/projected/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-kube-api-access-5llk8\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.429937 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.429795 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-maas-api-tls\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.532386 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.532257 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5llk8\" (UniqueName: \"kubernetes.io/projected/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-kube-api-access-5llk8\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.532573 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.532453 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-maas-api-tls\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.532663 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.532647 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-openshift-service-ca\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.532743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.532682 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-metrics-certs\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.533543 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.533519 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-openshift-service-ca\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.535232 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.535210 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-maas-api-tls\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.535506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.535483 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-metrics-certs\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.542183 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.542151 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llk8\" (UniqueName: \"kubernetes.io/projected/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-kube-api-access-5llk8\") pod \"maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.644142 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.641571 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:31.955124 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:57:31.955085 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7c8d4d_a878_413a_b8fa_b2bc8e8d6bd6.slice/crio-c254f0e0f17b1779bdbc5c55dd9395d8e44a8fdd73a5c8f90b2edafa6580a5a3 WatchSource:0}: Error finding container c254f0e0f17b1779bdbc5c55dd9395d8e44a8fdd73a5c8f90b2edafa6580a5a3: Status 404 returned error can't find the container with id c254f0e0f17b1779bdbc5c55dd9395d8e44a8fdd73a5c8f90b2edafa6580a5a3 Nov 03 10:57:31.955776 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:31.955742 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk"] Nov 03 10:57:32.949995 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:32.949959 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" event={"ID":"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6","Type":"ContainerStarted","Data":"2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6"} Nov 03 10:57:32.950459 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:32.950069 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:32.950459 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:32.950099 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" event={"ID":"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6","Type":"ContainerStarted","Data":"c254f0e0f17b1779bdbc5c55dd9395d8e44a8fdd73a5c8f90b2edafa6580a5a3"} Nov 03 10:57:32.972703 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:32.972634 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" podStartSLOduration=1.438691608 podStartE2EDuration="1.972615308s" totalImagesPullingTime="533.9237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:32.968517322 +0000 UTC m=+1730.953388619" watchObservedRunningTime="2026-11-03 10:57:32.972615308 +0000 UTC m=+1730.957486593" Nov 03 10:57:36.946010 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:36.945981 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 10:57:38.962918 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:38.962884 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:57:46.150315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.150265 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n"] Nov 03 10:57:46.154880 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.154860 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.161582 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.161540 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-836589-serving-cert\"" Nov 03 10:57:46.208543 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.208455 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n"] Nov 03 10:57:46.306849 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.305867 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.306849 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.305948 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzp7x\" (UniqueName: \"kubernetes.io/projected/ea69424b-3f91-47a6-b534-604ca3d09332-kube-api-access-lzp7x\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.306849 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.306021 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69424b-3f91-47a6-b534-604ca3d09332-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.306849 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.306128 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.407888 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.407786 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.407888 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.407864 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzp7x\" (UniqueName: \"kubernetes.io/projected/ea69424b-3f91-47a6-b534-604ca3d09332-kube-api-access-lzp7x\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.408138 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.407941 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69424b-3f91-47a6-b534-604ca3d09332-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.408138 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.408034 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.409255 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.409227 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69424b-3f91-47a6-b534-604ca3d09332-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.411858 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.411829 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.412185 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.412160 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.416677 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.416653 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzp7x\" (UniqueName: \"kubernetes.io/projected/ea69424b-3f91-47a6-b534-604ca3d09332-kube-api-access-lzp7x\") pod \"maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.489907 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.489863 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:46.776433 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:46.776384 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n"] Nov 03 10:57:46.776574 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:57:46.776536 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea69424b_3f91_47a6_b534_604ca3d09332.slice/crio-73398049ef1ebdae6fa5c4074e4c2617fcf1afda9acb3bb00767ab2c4ee19f3f WatchSource:0}: Error finding container 73398049ef1ebdae6fa5c4074e4c2617fcf1afda9acb3bb00767ab2c4ee19f3f: Status 404 returned error can't find the container with id 73398049ef1ebdae6fa5c4074e4c2617fcf1afda9acb3bb00767ab2c4ee19f3f Nov 03 10:57:47.080800 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:47.080765 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" event={"ID":"ea69424b-3f91-47a6-b534-604ca3d09332","Type":"ContainerStarted","Data":"73398049ef1ebdae6fa5c4074e4c2617fcf1afda9acb3bb00767ab2c4ee19f3f"} Nov 03 10:57:48.090180 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:48.090132 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" event={"ID":"ea69424b-3f91-47a6-b534-604ca3d09332","Type":"ContainerStarted","Data":"6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f"} Nov 03 10:57:48.090655 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:48.090216 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:48.108247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:48.108193 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" podStartSLOduration=1.6458909350000002 podStartE2EDuration="2.108179287s" totalImagesPullingTime="462.288352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:48.106470654 +0000 UTC m=+1746.091341940" watchObservedRunningTime="2026-11-03 10:57:48.108179287 +0000 UTC m=+1746.093050570" Nov 03 10:57:54.104054 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:57:54.103967 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 10:57:54.276187 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:57:54.276143 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:58:23.476506 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.476386 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk"] Nov 03 10:58:23.477381 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.477349 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" podUID="6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" containerName="maas-api" containerID="cri-o://2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6" gracePeriod=30 Nov 03 10:58:23.757423 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.757392 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:58:23.784108 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.784054 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-openshift-service-ca\") pod \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " Nov 03 10:58:23.784326 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.784193 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-kube-api-access-5llk8\" (UniqueName: \"kubernetes.io/projected/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-kube-api-access-5llk8\") pod \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " Nov 03 10:58:23.784480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.784336 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-maas-api-tls\") pod \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " Nov 03 10:58:23.784480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.784390 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-metrics-certs\") pod \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\" (UID: \"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6\") " Nov 03 10:58:23.784480 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.784445 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-openshift-service-ca" pod "6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" (UID: "6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:58:23.784980 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.784954 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:58:23.786729 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.786701 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-metrics-certs" pod "6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" (UID: "6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:58:23.787103 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.787074 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-kube-api-access-5llk8" pod "6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" (UID: "6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6"). InnerVolumeSpecName "kube-api-access-5llk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:58:23.787206 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.787136 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-maas-api-tls" pod "6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" (UID: "6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:58:23.886877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.886759 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:58:23.886877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.886816 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5llk8\" (UniqueName: \"kubernetes.io/projected/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-kube-api-access-5llk8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:58:23.886877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:23.886839 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:58:24.277319 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:58:24.277258 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:58:24.494120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:24.494028 5086 generic.go:356] "Generic (PLEG): container finished" podID="6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" containerID="2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6" exitCode=0 Nov 03 10:58:24.494120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:24.494082 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" Nov 03 10:58:24.494120 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:24.494107 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" event={"ID":"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6","Type":"ContainerDied","Data":"2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6"} Nov 03 10:58:24.494803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:24.494165 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk" event={"ID":"6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6","Type":"ContainerDied","Data":"c254f0e0f17b1779bdbc5c55dd9395d8e44a8fdd73a5c8f90b2edafa6580a5a3"} Nov 03 10:58:24.494803 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:24.494190 5086 scope.go:122] "RemoveContainer" containerID="2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6" Nov 03 10:58:24.505237 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:24.505216 5086 scope.go:122] "RemoveContainer" containerID="2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6" Nov 03 10:58:24.505558 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:58:24.505539 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6\": container with ID starting with 2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6 not found: ID does not exist" containerID="2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6" Nov 03 10:58:24.505622 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:24.505570 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6"} err="failed to get container status \"2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6\": rpc error: code = NotFound desc = could not find container \"2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6\": container with ID starting with 2037cd875671535cd6beebf9bc96f7523c9a3194d84481bfb9e9bbd693ee55a6 not found: ID does not exist" Nov 03 10:58:24.516548 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:24.516522 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk"] Nov 03 10:58:24.522729 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:24.522702 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3ba121-867dd857f5-ttkpk"] Nov 03 10:58:26.172199 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:26.172165 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" path="/var/lib/kubelet/pods/6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6/volumes" Nov 03 10:58:36.196384 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.196350 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4"] Nov 03 10:58:36.197093 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.197076 5086 cpu_manager.go:414] "containerMap: removing container" podUID="6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" containerName="maas-api" Nov 03 10:58:36.197145 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.197095 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" containerName="maas-api" Nov 03 10:58:36.197248 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.197238 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7c8d4d-a878-413a-b8fa-b2bc8e8d6bd6" containerName="maas-api" Nov 03 10:58:36.200995 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.200972 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.208798 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.208769 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b5a832\"/\"openshift-service-ca.crt\"" Nov 03 10:58:36.209079 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.209062 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b5a832\"/\"kube-root-ca.crt\"" Nov 03 10:58:36.209320 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.209286 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b5a832\"/\"default-dockercfg-8972d\"" Nov 03 10:58:36.209518 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.209495 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b5a832\"/\"noten-model-b5a832-kserve-self-signed-certs\"" Nov 03 10:58:36.211929 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.211909 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4"] Nov 03 10:58:36.285804 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.285762 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrn4\" (UniqueName: \"kubernetes.io/projected/00678448-2183-47d0-b1b5-c9e8fe6b6357-kube-api-access-wqrn4\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.286008 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.285854 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-dshm\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.286092 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.286010 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-tmp-dir\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.286092 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.286068 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-home\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.286220 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.286202 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00678448-2183-47d0-b1b5-c9e8fe6b6357-tls-certs\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.286259 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.286241 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-model-cache\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.392434 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.392390 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrn4\" (UniqueName: \"kubernetes.io/projected/00678448-2183-47d0-b1b5-c9e8fe6b6357-kube-api-access-wqrn4\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.397586 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.397544 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-dshm\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.397795 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.397647 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-tmp-dir\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.397795 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.397713 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-home\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.397938 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.397811 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00678448-2183-47d0-b1b5-c9e8fe6b6357-tls-certs\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.397938 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.397857 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-model-cache\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.398510 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.398481 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-model-cache\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.401725 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.401671 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-home\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.402577 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.402489 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-tmp-dir\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.409619 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.409540 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-dshm\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.410333 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.410288 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrn4\" (UniqueName: \"kubernetes.io/projected/00678448-2183-47d0-b1b5-c9e8fe6b6357-kube-api-access-wqrn4\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.412218 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.411777 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00678448-2183-47d0-b1b5-c9e8fe6b6357-tls-certs\") pod \"noten-model-b5a832-kserve-78495bd9df-rlhn4\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.547583 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.547479 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:36.843983 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:58:36.843949 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00678448_2183_47d0_b1b5_c9e8fe6b6357.slice/crio-f5b1fc25ffb98e7ce35bbf0857a0d0697084051fc334c981ca58bbe0ae20210a WatchSource:0}: Error finding container f5b1fc25ffb98e7ce35bbf0857a0d0697084051fc334c981ca58bbe0ae20210a: Status 404 returned error can't find the container with id f5b1fc25ffb98e7ce35bbf0857a0d0697084051fc334c981ca58bbe0ae20210a Nov 03 10:58:36.845049 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:36.845026 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4"] Nov 03 10:58:37.606584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:37.606543 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" event={"ID":"00678448-2183-47d0-b1b5-c9e8fe6b6357","Type":"ContainerStarted","Data":"6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f"} Nov 03 10:58:37.606584 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:37.606584 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" event={"ID":"00678448-2183-47d0-b1b5-c9e8fe6b6357","Type":"ContainerStarted","Data":"f5b1fc25ffb98e7ce35bbf0857a0d0697084051fc334c981ca58bbe0ae20210a"} Nov 03 10:58:37.622242 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:37.622184 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" podStartSLOduration=1.62217009 podStartE2EDuration="1.62217009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:58:37.621274949 +0000 UTC m=+1795.606146234" watchObservedRunningTime="2026-11-03 10:58:37.62217009 +0000 UTC m=+1795.607041373" Nov 03 10:58:38.547910 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:38.547876 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:58:42.934413 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:42.926353 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:58:42.938664 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:42.938635 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:58:42.957221 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:42.957196 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 10:58:42.962165 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:42.962144 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 10:58:45.277449 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:58:45.277402 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4"] Nov 03 10:58:54.275930 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:58:54.275889 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:59:00.306521 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.306450 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" podUID="00678448-2183-47d0-b1b5-c9e8fe6b6357" containerName="main" containerID="cri-o://6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f" gracePeriod=45 Nov 03 10:59:00.721837 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.721805 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:59:00.824315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.821944 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00678448-2183-47d0-b1b5-c9e8fe6b6357-tls-certs\" (UniqueName: \"kubernetes.io/secret/00678448-2183-47d0-b1b5-c9e8fe6b6357-tls-certs\") pod \"00678448-2183-47d0-b1b5-c9e8fe6b6357\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " Nov 03 10:59:00.824315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.822203 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-home\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-home\") pod \"00678448-2183-47d0-b1b5-c9e8fe6b6357\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " Nov 03 10:59:00.824315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.822310 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-dshm\") pod \"00678448-2183-47d0-b1b5-c9e8fe6b6357\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " Nov 03 10:59:00.824315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.822338 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-tmp-dir\") pod \"00678448-2183-47d0-b1b5-c9e8fe6b6357\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " Nov 03 10:59:00.824315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.822387 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00678448-2183-47d0-b1b5-c9e8fe6b6357-kube-api-access-wqrn4\" (UniqueName: \"kubernetes.io/projected/00678448-2183-47d0-b1b5-c9e8fe6b6357-kube-api-access-wqrn4\") pod \"00678448-2183-47d0-b1b5-c9e8fe6b6357\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " Nov 03 10:59:00.824315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.822414 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-model-cache\") pod \"00678448-2183-47d0-b1b5-c9e8fe6b6357\" (UID: \"00678448-2183-47d0-b1b5-c9e8fe6b6357\") " Nov 03 10:59:00.824315 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.823408 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-model-cache" pod "00678448-2183-47d0-b1b5-c9e8fe6b6357" (UID: "00678448-2183-47d0-b1b5-c9e8fe6b6357"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:59:00.839317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.826640 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-home" pod "00678448-2183-47d0-b1b5-c9e8fe6b6357" (UID: "00678448-2183-47d0-b1b5-c9e8fe6b6357"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:59:00.839317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.826741 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-tmp-dir" pod "00678448-2183-47d0-b1b5-c9e8fe6b6357" (UID: "00678448-2183-47d0-b1b5-c9e8fe6b6357"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:59:00.839317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.829223 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00678448-2183-47d0-b1b5-c9e8fe6b6357-kube-api-access-wqrn4" pod "00678448-2183-47d0-b1b5-c9e8fe6b6357" (UID: "00678448-2183-47d0-b1b5-c9e8fe6b6357"). InnerVolumeSpecName "kube-api-access-wqrn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:59:00.839317 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.829916 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-dshm" pod "00678448-2183-47d0-b1b5-c9e8fe6b6357" (UID: "00678448-2183-47d0-b1b5-c9e8fe6b6357"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:59:00.859609 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.859543 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00678448-2183-47d0-b1b5-c9e8fe6b6357-tls-certs" pod "00678448-2183-47d0-b1b5-c9e8fe6b6357" (UID: "00678448-2183-47d0-b1b5-c9e8fe6b6357"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:59:00.870580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.869561 5086 generic.go:356] "Generic (PLEG): container finished" podID="00678448-2183-47d0-b1b5-c9e8fe6b6357" containerID="6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f" exitCode=0 Nov 03 10:59:00.870580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.869621 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" Nov 03 10:59:00.870580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.869768 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" event={"ID":"00678448-2183-47d0-b1b5-c9e8fe6b6357","Type":"ContainerDied","Data":"6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f"} Nov 03 10:59:00.870580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.869812 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4" event={"ID":"00678448-2183-47d0-b1b5-c9e8fe6b6357","Type":"ContainerDied","Data":"f5b1fc25ffb98e7ce35bbf0857a0d0697084051fc334c981ca58bbe0ae20210a"} Nov 03 10:59:00.870580 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.869836 5086 scope.go:122] "RemoveContainer" containerID="6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f" Nov 03 10:59:00.886112 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.886053 5086 scope.go:122] "RemoveContainer" containerID="6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f" Nov 03 10:59:00.886511 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:00.886490 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f\": container with ID starting with 6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f not found: ID does not exist" containerID="6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f" Nov 03 10:59:00.886662 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.886643 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f"} err="failed to get container status \"6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f\": rpc error: code = NotFound desc = could not find container \"6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f\": container with ID starting with 6a403f18159ecf7372dddda24be1ce55d61d816ae6c109e92c6a76c52a752a3f not found: ID does not exist" Nov 03 10:59:00.900755 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.900726 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4"] Nov 03 10:59:00.904259 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.904223 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-b5a832/noten-model-b5a832-kserve-78495bd9df-rlhn4"] Nov 03 10:59:00.924493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.924417 5086 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:59:00.924493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.924443 5086 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:59:00.924493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.924452 5086 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:59:00.924493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.924462 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqrn4\" (UniqueName: \"kubernetes.io/projected/00678448-2183-47d0-b1b5-c9e8fe6b6357-kube-api-access-wqrn4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:59:00.924493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.924474 5086 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00678448-2183-47d0-b1b5-c9e8fe6b6357-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:59:00.924493 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:00.924484 5086 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00678448-2183-47d0-b1b5-c9e8fe6b6357-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:59:02.080738 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.080704 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-2136b0"] Nov 03 10:59:02.081642 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.081617 5086 cpu_manager.go:414] "containerMap: removing container" podUID="00678448-2183-47d0-b1b5-c9e8fe6b6357" containerName="main" Nov 03 10:59:02.081642 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.081641 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00678448-2183-47d0-b1b5-c9e8fe6b6357" containerName="main" Nov 03 10:59:02.081877 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.081861 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="00678448-2183-47d0-b1b5-c9e8fe6b6357" containerName="main" Nov 03 10:59:02.087217 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.087189 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-2136b0" Nov 03 10:59:02.090878 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.090851 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5zm6k\"" Nov 03 10:59:02.091410 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.091380 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-2136b0"] Nov 03 10:59:02.143838 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.143803 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwtj\" (UniqueName: \"kubernetes.io/projected/8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd-kube-api-access-lnwtj\") pod \"test-curl-7600-2136b0\" (UID: \"8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd\") " pod="opendatahub/test-curl-7600-2136b0" Nov 03 10:59:02.172134 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.172101 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00678448-2183-47d0-b1b5-c9e8fe6b6357" path="/var/lib/kubelet/pods/00678448-2183-47d0-b1b5-c9e8fe6b6357/volumes" Nov 03 10:59:02.245729 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.245697 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwtj\" (UniqueName: \"kubernetes.io/projected/8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd-kube-api-access-lnwtj\") pod \"test-curl-7600-2136b0\" (UID: \"8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd\") " pod="opendatahub/test-curl-7600-2136b0" Nov 03 10:59:02.253498 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.253469 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwtj\" (UniqueName: \"kubernetes.io/projected/8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd-kube-api-access-lnwtj\") pod \"test-curl-7600-2136b0\" (UID: \"8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd\") " pod="opendatahub/test-curl-7600-2136b0" Nov 03 10:59:02.407245 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.407153 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-2136b0" Nov 03 10:59:02.689408 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.689370 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-2136b0"] Nov 03 10:59:02.690764 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:59:02.690731 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d89bed1_ec9a_4f1c_bccf_c7e451aa35bd.slice/crio-f19ac425787ed768d0db88c71966181d19f02c6bc7d0c042fc4d9ff1550e0436 WatchSource:0}: Error finding container f19ac425787ed768d0db88c71966181d19f02c6bc7d0c042fc4d9ff1550e0436: Status 404 returned error can't find the container with id f19ac425787ed768d0db88c71966181d19f02c6bc7d0c042fc4d9ff1550e0436 Nov 03 10:59:02.887452 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:02.887416 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-2136b0" event={"ID":"8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd","Type":"ContainerStarted","Data":"f19ac425787ed768d0db88c71966181d19f02c6bc7d0c042fc4d9ff1550e0436"} Nov 03 10:59:04.908428 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:04.908395 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-2136b0" event={"ID":"8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd","Type":"ContainerStarted","Data":"6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7"} Nov 03 10:59:04.922971 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:04.922918 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-2136b0" podStartSLOduration=1.821726124 podStartE2EDuration="2.922903506s" totalImagesPullingTime="1.101177382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:04.921887463 +0000 UTC m=+1822.906758727" watchObservedRunningTime="2026-11-03 10:59:04.922903506 +0000 UTC m=+1822.907774789" Nov 03 10:59:05.564875 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.564839 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-2136b0"] Nov 03 10:59:05.568612 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.568581 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-2136b0"] Nov 03 10:59:05.727873 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.727842 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-9feb67"] Nov 03 10:59:05.728601 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.728582 5086 cpu_manager.go:414] "containerMap: removing container" podUID="8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd" containerName="test-curl-7600-2136b0" Nov 03 10:59:05.728601 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.728601 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd" containerName="test-curl-7600-2136b0" Nov 03 10:59:05.728726 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.728699 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd" containerName="test-curl-7600-2136b0" Nov 03 10:59:05.732746 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.732712 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-9feb67" Nov 03 10:59:05.735529 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.735496 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-9feb67"] Nov 03 10:59:05.805785 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.805752 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jhq\" (UniqueName: \"kubernetes.io/projected/4a9a6e3d-5703-4a39-b09d-bfe666d9539e-kube-api-access-b4jhq\") pod \"test-curl-7600-9feb67\" (UID: \"4a9a6e3d-5703-4a39-b09d-bfe666d9539e\") " pod="opendatahub/test-curl-7600-9feb67" Nov 03 10:59:05.906742 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.906702 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jhq\" (UniqueName: \"kubernetes.io/projected/4a9a6e3d-5703-4a39-b09d-bfe666d9539e-kube-api-access-b4jhq\") pod \"test-curl-7600-9feb67\" (UID: \"4a9a6e3d-5703-4a39-b09d-bfe666d9539e\") " pod="opendatahub/test-curl-7600-9feb67" Nov 03 10:59:05.914744 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:05.914711 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jhq\" (UniqueName: \"kubernetes.io/projected/4a9a6e3d-5703-4a39-b09d-bfe666d9539e-kube-api-access-b4jhq\") pod \"test-curl-7600-9feb67\" (UID: \"4a9a6e3d-5703-4a39-b09d-bfe666d9539e\") " pod="opendatahub/test-curl-7600-9feb67" Nov 03 10:59:06.054608 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:06.054572 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-9feb67" Nov 03 10:59:06.341491 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:06.341451 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-9feb67"] Nov 03 10:59:06.341947 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:59:06.341922 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a9a6e3d_5703_4a39_b09d_bfe666d9539e.slice/crio-024a63e3b69e8dfb88f740926fbb217aa32e05a940e2cb0b470b7fe1cff3ccbb WatchSource:0}: Error finding container 024a63e3b69e8dfb88f740926fbb217aa32e05a940e2cb0b470b7fe1cff3ccbb: Status 404 returned error can't find the container with id 024a63e3b69e8dfb88f740926fbb217aa32e05a940e2cb0b470b7fe1cff3ccbb Nov 03 10:59:06.924268 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:06.924227 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-9feb67" event={"ID":"4a9a6e3d-5703-4a39-b09d-bfe666d9539e","Type":"ContainerStarted","Data":"2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139"} Nov 03 10:59:06.924768 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:06.924282 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-9feb67" event={"ID":"4a9a6e3d-5703-4a39-b09d-bfe666d9539e","Type":"ContainerStarted","Data":"024a63e3b69e8dfb88f740926fbb217aa32e05a940e2cb0b470b7fe1cff3ccbb"} Nov 03 10:59:06.924768 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:06.924356 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-2136b0" podUID="8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd" containerName="test-curl-7600-2136b0" containerID="cri-o://6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7" gracePeriod=2 Nov 03 10:59:06.942217 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:06.942166 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-9feb67" podStartSLOduration=1.9421524570000002 podStartE2EDuration="1.942152457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:06.938882628 +0000 UTC m=+1824.923753922" watchObservedRunningTime="2026-11-03 10:59:06.942152457 +0000 UTC m=+1824.927023740" Nov 03 10:59:07.649287 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:07.649250 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-9feb67"] Nov 03 10:59:07.655543 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:07.655509 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-9feb67"] Nov 03 10:59:08.945783 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:08.945723 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-9feb67" podUID="4a9a6e3d-5703-4a39-b09d-bfe666d9539e" containerName="test-curl-7600-9feb67" containerID="cri-o://2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139" gracePeriod=2 Nov 03 10:59:09.180568 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:09.180504 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-2136b0" Nov 03 10:59:09.275717 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:09.275681 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd-kube-api-access-lnwtj\" (UniqueName: \"kubernetes.io/projected/8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd-kube-api-access-lnwtj\") pod \"8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd\" (UID: \"8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd\") " Nov 03 10:59:09.277897 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:09.277865 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd-kube-api-access-lnwtj" pod "8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd" (UID: "8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd"). InnerVolumeSpecName "kube-api-access-lnwtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:59:09.378990 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:09.378843 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnwtj\" (UniqueName: \"kubernetes.io/projected/8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd-kube-api-access-lnwtj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:59:09.954212 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:09.954175 5086 generic.go:356] "Generic (PLEG): container finished" podID="8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd" containerID="6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7" exitCode=137 Nov 03 10:59:09.954684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:09.954224 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-2136b0" Nov 03 10:59:09.954684 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:09.954243 5086 scope.go:122] "RemoveContainer" containerID="6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7" Nov 03 10:59:09.964383 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:09.964364 5086 scope.go:122] "RemoveContainer" containerID="6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7" Nov 03 10:59:09.964663 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:09.964646 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7\": container with ID starting with 6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7 not found: ID does not exist" containerID="6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7" Nov 03 10:59:09.964716 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:09.964672 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7"} err="failed to get container status \"6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7\": rpc error: code = NotFound desc = could not find container \"6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7\": container with ID starting with 6e8baa7bb017ac0cd6ac4aa8d5e05d444f321f369278328099d4238f418f24e7 not found: ID does not exist" Nov 03 10:59:09.964882 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:09.964863 5086 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-2136b0\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd" pod="opendatahub/test-curl-7600-2136b0" Nov 03 10:59:10.184786 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:10.184753 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd" path="/var/lib/kubelet/pods/8d89bed1-ec9a-4f1c-bccf-c7e451aa35bd/volumes" Nov 03 10:59:11.167544 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:11.167515 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-9feb67" Nov 03 10:59:11.306179 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:11.306089 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a9a6e3d-5703-4a39-b09d-bfe666d9539e-kube-api-access-b4jhq\" (UniqueName: \"kubernetes.io/projected/4a9a6e3d-5703-4a39-b09d-bfe666d9539e-kube-api-access-b4jhq\") pod \"4a9a6e3d-5703-4a39-b09d-bfe666d9539e\" (UID: \"4a9a6e3d-5703-4a39-b09d-bfe666d9539e\") " Nov 03 10:59:11.308507 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:11.308478 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9a6e3d-5703-4a39-b09d-bfe666d9539e-kube-api-access-b4jhq" pod "4a9a6e3d-5703-4a39-b09d-bfe666d9539e" (UID: "4a9a6e3d-5703-4a39-b09d-bfe666d9539e"). InnerVolumeSpecName "kube-api-access-b4jhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:59:11.408895 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:11.408858 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4jhq\" (UniqueName: \"kubernetes.io/projected/4a9a6e3d-5703-4a39-b09d-bfe666d9539e-kube-api-access-b4jhq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 10:59:11.974061 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:11.974021 5086 generic.go:356] "Generic (PLEG): container finished" podID="4a9a6e3d-5703-4a39-b09d-bfe666d9539e" containerID="2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139" exitCode=137 Nov 03 10:59:11.974332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:11.974072 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-9feb67" Nov 03 10:59:11.974332 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:11.974128 5086 scope.go:122] "RemoveContainer" containerID="2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139" Nov 03 10:59:11.986876 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:11.986728 5086 scope.go:122] "RemoveContainer" containerID="2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139" Nov 03 10:59:11.986942 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:11.986891 5086 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-9feb67\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="4a9a6e3d-5703-4a39-b09d-bfe666d9539e" pod="opendatahub/test-curl-7600-9feb67" Nov 03 10:59:11.987106 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:11.987081 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139\": container with ID starting with 2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139 not found: ID does not exist" containerID="2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139" Nov 03 10:59:11.987178 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:11.987116 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139"} err="failed to get container status \"2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139\": rpc error: code = NotFound desc = could not find container \"2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139\": container with ID starting with 2b22779af9ca6ba611288ec416c5367258b1d0bb843df21c6a39cd4ed9ddc139 not found: ID does not exist" Nov 03 10:59:12.169366 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:12.169328 5086 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-9feb67\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="4a9a6e3d-5703-4a39-b09d-bfe666d9539e" pod="opendatahub/test-curl-7600-9feb67" Nov 03 10:59:12.172346 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:12.172321 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a9a6e3d-5703-4a39-b09d-bfe666d9539e" path="/var/lib/kubelet/pods/4a9a6e3d-5703-4a39-b09d-bfe666d9539e/volumes" Nov 03 10:59:21.538944 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.538906 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl"] Nov 03 10:59:21.539677 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.539659 5086 cpu_manager.go:414] "containerMap: removing container" podUID="4a9a6e3d-5703-4a39-b09d-bfe666d9539e" containerName="test-curl-7600-9feb67" Nov 03 10:59:21.539677 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.539676 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a9a6e3d-5703-4a39-b09d-bfe666d9539e" containerName="test-curl-7600-9feb67" Nov 03 10:59:21.539822 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.539795 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a9a6e3d-5703-4a39-b09d-bfe666d9539e" containerName="test-curl-7600-9feb67" Nov 03 10:59:21.544905 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.544877 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.560327 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.557619 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-9c53b2-serving-cert\"" Nov 03 10:59:21.564247 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.563529 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl"] Nov 03 10:59:21.663350 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.663281 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.663727 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.663684 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d277c8b6-7753-4614-959d-e79d38a77fc4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.664082 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.664062 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjqgv\" (UniqueName: \"kubernetes.io/projected/d277c8b6-7753-4614-959d-e79d38a77fc4-kube-api-access-sjqgv\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.664527 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.664502 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.766557 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.766522 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.766743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.766581 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d277c8b6-7753-4614-959d-e79d38a77fc4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.766743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.766680 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjqgv\" (UniqueName: \"kubernetes.io/projected/d277c8b6-7753-4614-959d-e79d38a77fc4-kube-api-access-sjqgv\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.766743 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.766703 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.766871 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:21.766850 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-serving-cert: secret "maas-api-e2e-shared-a-w0-9c53b2-serving-cert" not found Nov 03 10:59:21.766911 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:21.766906 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls podName:d277c8b6-7753-4614-959d-e79d38a77fc4 nodeName:}" failed. No retries permitted until 2026-11-03 10:59:22.266890235 +0000 UTC m=+1840.251761508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls") pod "maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" (UID: "d277c8b6-7753-4614-959d-e79d38a77fc4") : secret "maas-api-e2e-shared-a-w0-9c53b2-serving-cert" not found Nov 03 10:59:21.767476 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.767447 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d277c8b6-7753-4614-959d-e79d38a77fc4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.769192 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.769169 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:21.774821 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:21.774795 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjqgv\" (UniqueName: \"kubernetes.io/projected/d277c8b6-7753-4614-959d-e79d38a77fc4-kube-api-access-sjqgv\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:22.274201 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:22.274161 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:22.277229 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:22.277188 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:22.485139 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:22.485098 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:22.818070 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:22.818043 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl"] Nov 03 10:59:22.819204 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:59:22.819175 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd277c8b6_7753_4614_959d_e79d38a77fc4.slice/crio-9207cfc37fff402bcdb8c66b4015d7837a50634fcdfcb87e824194441b82e47c WatchSource:0}: Error finding container 9207cfc37fff402bcdb8c66b4015d7837a50634fcdfcb87e824194441b82e47c: Status 404 returned error can't find the container with id 9207cfc37fff402bcdb8c66b4015d7837a50634fcdfcb87e824194441b82e47c Nov 03 10:59:23.080665 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:23.080627 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" event={"ID":"d277c8b6-7753-4614-959d-e79d38a77fc4","Type":"ContainerStarted","Data":"9207cfc37fff402bcdb8c66b4015d7837a50634fcdfcb87e824194441b82e47c"} Nov 03 10:59:24.089557 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:24.089509 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" event={"ID":"d277c8b6-7753-4614-959d-e79d38a77fc4","Type":"ContainerStarted","Data":"bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077"} Nov 03 10:59:24.089984 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:24.089588 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:24.108366 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:24.108316 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" podStartSLOduration=2.577281567 podStartE2EDuration="3.108289356s" totalImagesPullingTime="531.007789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:24.105431306 +0000 UTC m=+1842.090302601" watchObservedRunningTime="2026-11-03 10:59:24.108289356 +0000 UTC m=+1842.093160639" Nov 03 10:59:24.307481 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:24.307440 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:59:30.103940 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:30.103906 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 10:59:43.821092 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:43.821058 5086 scope.go:122] "RemoveContainer" containerID="dd0762fda1f194244c854e008a5283e70c1092c06c72f0f33275e0fab49f8618" Nov 03 10:59:53.068658 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.068620 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9"] Nov 03 10:59:53.075736 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.075709 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.085884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.083712 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.085884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.083828 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptsb\" (UniqueName: \"kubernetes.io/projected/0063172a-cdd0-4c8d-b0c9-33f77c87c769-kube-api-access-8ptsb\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.085884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.083989 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063172a-cdd0-4c8d-b0c9-33f77c87c769-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.085884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.084038 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.085884 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.084646 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-ef1662-serving-cert\"" Nov 03 10:59:53.088053 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.088027 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9"] Nov 03 10:59:53.202200 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.202157 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.202508 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.202340 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.202508 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.202454 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptsb\" (UniqueName: \"kubernetes.io/projected/0063172a-cdd0-4c8d-b0c9-33f77c87c769-kube-api-access-8ptsb\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.202766 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.202744 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063172a-cdd0-4c8d-b0c9-33f77c87c769-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.202915 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:53.202896 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-serving-cert: secret "maas-api-e2e-shared-b-w0-ef1662-serving-cert" not found Nov 03 10:59:53.203180 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:53.203059 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls podName:0063172a-cdd0-4c8d-b0c9-33f77c87c769 nodeName:}" failed. No retries permitted until 2026-11-03 10:59:53.703033824 +0000 UTC m=+1871.687905099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls") pod "maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" (UID: "0063172a-cdd0-4c8d-b0c9-33f77c87c769") : secret "maas-api-e2e-shared-b-w0-ef1662-serving-cert" not found Nov 03 10:59:53.216051 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.215143 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063172a-cdd0-4c8d-b0c9-33f77c87c769-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.237935 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.233365 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptsb\" (UniqueName: \"kubernetes.io/projected/0063172a-cdd0-4c8d-b0c9-33f77c87c769-kube-api-access-8ptsb\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.248228 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.241777 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.714563 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.714512 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:53.734934 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:53.734883 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:54.006860 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:54.006760 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:54.275882 ip-10-0-139-90 kubenswrapper[5086]: E1103 10:59:54.275784 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 10:59:54.290836 ip-10-0-139-90 kubenswrapper[5086]: W1103 10:59:54.290809 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0063172a_cdd0_4c8d_b0c9_33f77c87c769.slice/crio-d4944b890fa85a12e9bcecbe08add095e1786ac20b42af46ecdfc14f87f65db2 WatchSource:0}: Error finding container d4944b890fa85a12e9bcecbe08add095e1786ac20b42af46ecdfc14f87f65db2: Status 404 returned error can't find the container with id d4944b890fa85a12e9bcecbe08add095e1786ac20b42af46ecdfc14f87f65db2 Nov 03 10:59:54.291114 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:54.291092 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9"] Nov 03 10:59:54.420136 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:54.420097 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" event={"ID":"0063172a-cdd0-4c8d-b0c9-33f77c87c769","Type":"ContainerStarted","Data":"d4944b890fa85a12e9bcecbe08add095e1786ac20b42af46ecdfc14f87f65db2"} Nov 03 10:59:55.430004 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:55.429965 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" event={"ID":"0063172a-cdd0-4c8d-b0c9-33f77c87c769","Type":"ContainerStarted","Data":"800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c"} Nov 03 10:59:55.430530 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:55.430060 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 10:59:55.471058 ip-10-0-139-90 kubenswrapper[5086]: I1103 10:59:55.470989 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" podStartSLOduration=2.04773419 podStartE2EDuration="2.470965701s" totalImagesPullingTime="423.231511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:55.467550798 +0000 UTC m=+1873.452422082" watchObservedRunningTime="2026-11-03 10:59:55.470965701 +0000 UTC m=+1873.455836996" Nov 03 11:00:01.442692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:01.442663 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 11:00:09.352592 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:09.352556 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-7b8f81"] Nov 03 11:00:09.357236 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:09.357206 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-7b8f81" Nov 03 11:00:09.360650 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:09.360627 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5zm6k\"" Nov 03 11:00:09.362700 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:09.362673 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-7b8f81"] Nov 03 11:00:09.473200 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:09.473165 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jln5f\" (UniqueName: \"kubernetes.io/projected/71102fa3-95ee-41c7-a59c-e8620c68a439-kube-api-access-jln5f\") pod \"test-curl-7600-7b8f81\" (UID: \"71102fa3-95ee-41c7-a59c-e8620c68a439\") " pod="opendatahub/test-curl-7600-7b8f81" Nov 03 11:00:09.574265 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:09.574232 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jln5f\" (UniqueName: \"kubernetes.io/projected/71102fa3-95ee-41c7-a59c-e8620c68a439-kube-api-access-jln5f\") pod \"test-curl-7600-7b8f81\" (UID: \"71102fa3-95ee-41c7-a59c-e8620c68a439\") " pod="opendatahub/test-curl-7600-7b8f81" Nov 03 11:00:09.581605 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:09.581581 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jln5f\" (UniqueName: \"kubernetes.io/projected/71102fa3-95ee-41c7-a59c-e8620c68a439-kube-api-access-jln5f\") pod \"test-curl-7600-7b8f81\" (UID: \"71102fa3-95ee-41c7-a59c-e8620c68a439\") " pod="opendatahub/test-curl-7600-7b8f81" Nov 03 11:00:09.673005 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:09.672934 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-7b8f81" Nov 03 11:00:09.950633 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:09.950602 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-7b8f81"] Nov 03 11:00:09.952485 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:00:09.952452 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71102fa3_95ee_41c7_a59c_e8620c68a439.slice/crio-18095290ea359abdee21302fdd156a34147db10148bd30c0d30c0e41912c1c7f WatchSource:0}: Error finding container 18095290ea359abdee21302fdd156a34147db10148bd30c0d30c0e41912c1c7f: Status 404 returned error can't find the container with id 18095290ea359abdee21302fdd156a34147db10148bd30c0d30c0e41912c1c7f Nov 03 11:00:10.585878 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:10.585832 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-7b8f81" event={"ID":"71102fa3-95ee-41c7-a59c-e8620c68a439","Type":"ContainerStarted","Data":"04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a"} Nov 03 11:00:10.586466 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:10.585898 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-7b8f81" event={"ID":"71102fa3-95ee-41c7-a59c-e8620c68a439","Type":"ContainerStarted","Data":"18095290ea359abdee21302fdd156a34147db10148bd30c0d30c0e41912c1c7f"} Nov 03 11:00:10.603312 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:10.603248 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-7b8f81" podStartSLOduration=1.603234151 podStartE2EDuration="1.603234151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:10.599243345 +0000 UTC m=+1888.584114629" watchObservedRunningTime="2026-11-03 11:00:10.603234151 +0000 UTC m=+1888.588105434" Nov 03 11:00:11.320909 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.320876 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-7b8f81"] Nov 03 11:00:11.326484 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.326427 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-7b8f81"] Nov 03 11:00:11.485353 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.485314 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-1f367c"] Nov 03 11:00:11.486436 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.486408 5086 cpu_manager.go:414] "containerMap: removing container" podUID="71102fa3-95ee-41c7-a59c-e8620c68a439" containerName="test-curl-7600-7b8f81" Nov 03 11:00:11.486436 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.486433 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71102fa3-95ee-41c7-a59c-e8620c68a439" containerName="test-curl-7600-7b8f81" Nov 03 11:00:11.486628 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.486571 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="71102fa3-95ee-41c7-a59c-e8620c68a439" containerName="test-curl-7600-7b8f81" Nov 03 11:00:11.490399 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.490371 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-1f367c" Nov 03 11:00:11.495644 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.495613 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-1f367c"] Nov 03 11:00:11.607177 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.607092 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69sq5\" (UniqueName: \"kubernetes.io/projected/c2ab46d1-5b76-411a-90fc-fe7aab4f837f-kube-api-access-69sq5\") pod \"test-curl-7600-1f367c\" (UID: \"c2ab46d1-5b76-411a-90fc-fe7aab4f837f\") " pod="opendatahub/test-curl-7600-1f367c" Nov 03 11:00:11.708533 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.708492 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69sq5\" (UniqueName: \"kubernetes.io/projected/c2ab46d1-5b76-411a-90fc-fe7aab4f837f-kube-api-access-69sq5\") pod \"test-curl-7600-1f367c\" (UID: \"c2ab46d1-5b76-411a-90fc-fe7aab4f837f\") " pod="opendatahub/test-curl-7600-1f367c" Nov 03 11:00:11.716403 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.716373 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69sq5\" (UniqueName: \"kubernetes.io/projected/c2ab46d1-5b76-411a-90fc-fe7aab4f837f-kube-api-access-69sq5\") pod \"test-curl-7600-1f367c\" (UID: \"c2ab46d1-5b76-411a-90fc-fe7aab4f837f\") " pod="opendatahub/test-curl-7600-1f367c" Nov 03 11:00:11.816082 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:11.816051 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-1f367c" Nov 03 11:00:12.130485 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:12.130456 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-1f367c"] Nov 03 11:00:12.131467 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:00:12.131435 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ab46d1_5b76_411a_90fc_fe7aab4f837f.slice/crio-c92d227daadf4e832dc3911059b2ff6eee5fd949c9f63d3a228a8d3447e4528b WatchSource:0}: Error finding container c92d227daadf4e832dc3911059b2ff6eee5fd949c9f63d3a228a8d3447e4528b: Status 404 returned error can't find the container with id c92d227daadf4e832dc3911059b2ff6eee5fd949c9f63d3a228a8d3447e4528b Nov 03 11:00:12.604565 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:12.604510 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-1f367c" event={"ID":"c2ab46d1-5b76-411a-90fc-fe7aab4f837f","Type":"ContainerStarted","Data":"7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac"} Nov 03 11:00:12.604774 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:12.604594 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-1f367c" event={"ID":"c2ab46d1-5b76-411a-90fc-fe7aab4f837f","Type":"ContainerStarted","Data":"c92d227daadf4e832dc3911059b2ff6eee5fd949c9f63d3a228a8d3447e4528b"} Nov 03 11:00:12.604774 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:12.604657 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-7b8f81" podUID="71102fa3-95ee-41c7-a59c-e8620c68a439" containerName="test-curl-7600-7b8f81" containerID="cri-o://04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a" gracePeriod=2 Nov 03 11:00:12.619926 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:12.619877 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-1f367c" podStartSLOduration=1.619863469 podStartE2EDuration="1.619863469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:12.617594628 +0000 UTC m=+1890.602465913" watchObservedRunningTime="2026-11-03 11:00:12.619863469 +0000 UTC m=+1890.604734752" Nov 03 11:00:13.325253 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.325220 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-1f367c"] Nov 03 11:00:13.330385 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.330352 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-1f367c"] Nov 03 11:00:13.506488 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.506453 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-53c342"] Nov 03 11:00:13.507164 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.507150 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c2ab46d1-5b76-411a-90fc-fe7aab4f837f" containerName="test-curl-7600-1f367c" Nov 03 11:00:13.507213 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.507165 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ab46d1-5b76-411a-90fc-fe7aab4f837f" containerName="test-curl-7600-1f367c" Nov 03 11:00:13.507329 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.507319 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ab46d1-5b76-411a-90fc-fe7aab4f837f" containerName="test-curl-7600-1f367c" Nov 03 11:00:13.512009 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.511983 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-53c342" Nov 03 11:00:13.514130 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.514104 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-53c342"] Nov 03 11:00:13.641922 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.641835 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krlgp\" (UniqueName: \"kubernetes.io/projected/f6f3275b-d95f-403f-ac0d-207602677186-kube-api-access-krlgp\") pod \"test-curl-7600-53c342\" (UID: \"f6f3275b-d95f-403f-ac0d-207602677186\") " pod="opendatahub/test-curl-7600-53c342" Nov 03 11:00:13.743431 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.743397 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krlgp\" (UniqueName: \"kubernetes.io/projected/f6f3275b-d95f-403f-ac0d-207602677186-kube-api-access-krlgp\") pod \"test-curl-7600-53c342\" (UID: \"f6f3275b-d95f-403f-ac0d-207602677186\") " pod="opendatahub/test-curl-7600-53c342" Nov 03 11:00:13.750784 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.750757 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krlgp\" (UniqueName: \"kubernetes.io/projected/f6f3275b-d95f-403f-ac0d-207602677186-kube-api-access-krlgp\") pod \"test-curl-7600-53c342\" (UID: \"f6f3275b-d95f-403f-ac0d-207602677186\") " pod="opendatahub/test-curl-7600-53c342" Nov 03 11:00:13.830454 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:13.830417 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-53c342" Nov 03 11:00:14.107237 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:14.107206 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-53c342"] Nov 03 11:00:14.107950 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:00:14.107925 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6f3275b_d95f_403f_ac0d_207602677186.slice/crio-f4004c2178146e53359c0d4c025a68d5776347f554600d6dd9cd7dcb9570d59e WatchSource:0}: Error finding container f4004c2178146e53359c0d4c025a68d5776347f554600d6dd9cd7dcb9570d59e: Status 404 returned error can't find the container with id f4004c2178146e53359c0d4c025a68d5776347f554600d6dd9cd7dcb9570d59e Nov 03 11:00:14.630289 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:14.629141 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-53c342" event={"ID":"f6f3275b-d95f-403f-ac0d-207602677186","Type":"ContainerStarted","Data":"f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3"} Nov 03 11:00:14.630289 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:14.629215 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-53c342" event={"ID":"f6f3275b-d95f-403f-ac0d-207602677186","Type":"ContainerStarted","Data":"f4004c2178146e53359c0d4c025a68d5776347f554600d6dd9cd7dcb9570d59e"} Nov 03 11:00:14.630289 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:14.629233 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-1f367c" podUID="c2ab46d1-5b76-411a-90fc-fe7aab4f837f" containerName="test-curl-7600-1f367c" containerID="cri-o://7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac" gracePeriod=2 Nov 03 11:00:14.656372 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:14.656289 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-53c342" podStartSLOduration=1.6562669730000001 podStartE2EDuration="1.656266973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:14.651159462 +0000 UTC m=+1892.636030746" watchObservedRunningTime="2026-11-03 11:00:14.656266973 +0000 UTC m=+1892.641138256" Nov 03 11:00:14.833998 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:14.833966 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-7b8f81" Nov 03 11:00:14.870586 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:14.870489 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71102fa3-95ee-41c7-a59c-e8620c68a439-kube-api-access-jln5f\" (UniqueName: \"kubernetes.io/projected/71102fa3-95ee-41c7-a59c-e8620c68a439-kube-api-access-jln5f\") pod \"71102fa3-95ee-41c7-a59c-e8620c68a439\" (UID: \"71102fa3-95ee-41c7-a59c-e8620c68a439\") " Nov 03 11:00:14.872797 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:14.872761 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71102fa3-95ee-41c7-a59c-e8620c68a439-kube-api-access-jln5f" pod "71102fa3-95ee-41c7-a59c-e8620c68a439" (UID: "71102fa3-95ee-41c7-a59c-e8620c68a439"). InnerVolumeSpecName "kube-api-access-jln5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:00:14.972567 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:14.972535 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jln5f\" (UniqueName: \"kubernetes.io/projected/71102fa3-95ee-41c7-a59c-e8620c68a439-kube-api-access-jln5f\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:00:15.486399 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.486364 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-53c342"] Nov 03 11:00:15.490557 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.490526 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-53c342"] Nov 03 11:00:15.637904 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.637870 5086 generic.go:356] "Generic (PLEG): container finished" podID="71102fa3-95ee-41c7-a59c-e8620c68a439" containerID="04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a" exitCode=137 Nov 03 11:00:15.638078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.637910 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-7b8f81" Nov 03 11:00:15.638078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.637970 5086 scope.go:122] "RemoveContainer" containerID="04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a" Nov 03 11:00:15.648337 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.648320 5086 scope.go:122] "RemoveContainer" containerID="04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a" Nov 03 11:00:15.648652 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:15.648632 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a\": container with ID starting with 04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a not found: ID does not exist" containerID="04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a" Nov 03 11:00:15.648721 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.648666 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a"} err="failed to get container status \"04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a\": rpc error: code = NotFound desc = could not find container \"04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a\": container with ID starting with 04be7674c0c8c8238ad4e8554b9ca7ed3f1f1ac0ccfa05b90b8b3c93d0b2817a not found: ID does not exist" Nov 03 11:00:15.649131 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:15.649109 5086 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-7b8f81\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="71102fa3-95ee-41c7-a59c-e8620c68a439" pod="opendatahub/test-curl-7600-7b8f81" Nov 03 11:00:15.664221 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.664195 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-b41f69"] Nov 03 11:00:15.665269 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.665232 5086 cpu_manager.go:414] "containerMap: removing container" podUID="f6f3275b-d95f-403f-ac0d-207602677186" containerName="test-curl-7600-53c342" Nov 03 11:00:15.665269 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.665257 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f3275b-d95f-403f-ac0d-207602677186" containerName="test-curl-7600-53c342" Nov 03 11:00:15.665476 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.665409 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f3275b-d95f-403f-ac0d-207602677186" containerName="test-curl-7600-53c342" Nov 03 11:00:15.686501 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.683668 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-b41f69" Nov 03 11:00:15.697109 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.697064 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-b41f69"] Nov 03 11:00:15.784568 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:15.782853 5086 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-7b8f81\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="71102fa3-95ee-41c7-a59c-e8620c68a439" pod="opendatahub/test-curl-7600-7b8f81" Nov 03 11:00:15.818567 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.816181 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999tx\" (UniqueName: \"kubernetes.io/projected/a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5-kube-api-access-999tx\") pod \"test-curl-7600-b41f69\" (UID: \"a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5\") " pod="opendatahub/test-curl-7600-b41f69" Nov 03 11:00:15.917897 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.917862 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999tx\" (UniqueName: \"kubernetes.io/projected/a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5-kube-api-access-999tx\") pod \"test-curl-7600-b41f69\" (UID: \"a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5\") " pod="opendatahub/test-curl-7600-b41f69" Nov 03 11:00:15.928900 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:15.928862 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-999tx\" (UniqueName: \"kubernetes.io/projected/a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5-kube-api-access-999tx\") pod \"test-curl-7600-b41f69\" (UID: \"a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5\") " pod="opendatahub/test-curl-7600-b41f69" Nov 03 11:00:16.120950 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.120914 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-b41f69" Nov 03 11:00:16.176059 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.176021 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71102fa3-95ee-41c7-a59c-e8620c68a439" path="/var/lib/kubelet/pods/71102fa3-95ee-41c7-a59c-e8620c68a439/volumes" Nov 03 11:00:16.406235 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.406199 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-b41f69"] Nov 03 11:00:16.406392 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:00:16.406284 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9b2d98a_5a92_4ab3_a2fa_5f138e9a47b5.slice/crio-dd74165ce56ec4125a62e42dc9f142bdc0734801726dd6a310da8cd06c3faaf7 WatchSource:0}: Error finding container dd74165ce56ec4125a62e42dc9f142bdc0734801726dd6a310da8cd06c3faaf7: Status 404 returned error can't find the container with id dd74165ce56ec4125a62e42dc9f142bdc0734801726dd6a310da8cd06c3faaf7 Nov 03 11:00:16.647339 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.647305 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-b41f69" event={"ID":"a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5","Type":"ContainerStarted","Data":"ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0"} Nov 03 11:00:16.647465 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.647374 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-b41f69" event={"ID":"a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5","Type":"ContainerStarted","Data":"dd74165ce56ec4125a62e42dc9f142bdc0734801726dd6a310da8cd06c3faaf7"} Nov 03 11:00:16.648437 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.648412 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-53c342" podUID="f6f3275b-d95f-403f-ac0d-207602677186" containerName="test-curl-7600-53c342" containerID="cri-o://f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3" gracePeriod=2 Nov 03 11:00:16.665103 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.665024 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-b41f69" podStartSLOduration=1.665011404 podStartE2EDuration="1.665011404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:16.661453609 +0000 UTC m=+1894.646324893" watchObservedRunningTime="2026-11-03 11:00:16.665011404 +0000 UTC m=+1894.649882686" Nov 03 11:00:16.860792 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.860769 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-1f367c" Nov 03 11:00:16.937047 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.936973 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ab46d1-5b76-411a-90fc-fe7aab4f837f-kube-api-access-69sq5\" (UniqueName: \"kubernetes.io/projected/c2ab46d1-5b76-411a-90fc-fe7aab4f837f-kube-api-access-69sq5\") pod \"c2ab46d1-5b76-411a-90fc-fe7aab4f837f\" (UID: \"c2ab46d1-5b76-411a-90fc-fe7aab4f837f\") " Nov 03 11:00:16.939041 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:16.939010 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ab46d1-5b76-411a-90fc-fe7aab4f837f-kube-api-access-69sq5" pod "c2ab46d1-5b76-411a-90fc-fe7aab4f837f" (UID: "c2ab46d1-5b76-411a-90fc-fe7aab4f837f"). InnerVolumeSpecName "kube-api-access-69sq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:00:17.039616 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:17.039165 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69sq5\" (UniqueName: \"kubernetes.io/projected/c2ab46d1-5b76-411a-90fc-fe7aab4f837f-kube-api-access-69sq5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:00:17.402560 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:17.402523 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-b41f69"] Nov 03 11:00:17.406243 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:17.406212 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-b41f69"] Nov 03 11:00:17.656708 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:17.656622 5086 generic.go:356] "Generic (PLEG): container finished" podID="c2ab46d1-5b76-411a-90fc-fe7aab4f837f" containerID="7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac" exitCode=137 Nov 03 11:00:17.656708 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:17.656664 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-1f367c" Nov 03 11:00:17.656945 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:17.656720 5086 scope.go:122] "RemoveContainer" containerID="7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac" Nov 03 11:00:17.667082 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:17.667065 5086 scope.go:122] "RemoveContainer" containerID="7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac" Nov 03 11:00:17.667513 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:17.667396 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac\": container with ID starting with 7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac not found: ID does not exist" containerID="7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac" Nov 03 11:00:17.667513 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:17.667422 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac"} err="failed to get container status \"7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac\": rpc error: code = NotFound desc = could not find container \"7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac\": container with ID starting with 7d458305ecdb0a3134b65e58ce2d1b643850803925848b5400723ddf8911e2ac not found: ID does not exist" Nov 03 11:00:17.667802 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:17.667780 5086 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-1f367c\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="c2ab46d1-5b76-411a-90fc-fe7aab4f837f" pod="opendatahub/test-curl-7600-1f367c" Nov 03 11:00:18.172540 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:18.172507 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ab46d1-5b76-411a-90fc-fe7aab4f837f" path="/var/lib/kubelet/pods/c2ab46d1-5b76-411a-90fc-fe7aab4f837f/volumes" Nov 03 11:00:18.667732 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:18.667677 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-b41f69" podUID="a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5" containerName="test-curl-7600-b41f69" containerID="cri-o://ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0" gracePeriod=2 Nov 03 11:00:18.865495 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:18.865470 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-53c342" Nov 03 11:00:18.976227 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:18.976144 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f3275b-d95f-403f-ac0d-207602677186-kube-api-access-krlgp\" (UniqueName: \"kubernetes.io/projected/f6f3275b-d95f-403f-ac0d-207602677186-kube-api-access-krlgp\") pod \"f6f3275b-d95f-403f-ac0d-207602677186\" (UID: \"f6f3275b-d95f-403f-ac0d-207602677186\") " Nov 03 11:00:18.978643 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:18.978609 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f3275b-d95f-403f-ac0d-207602677186-kube-api-access-krlgp" pod "f6f3275b-d95f-403f-ac0d-207602677186" (UID: "f6f3275b-d95f-403f-ac0d-207602677186"). InnerVolumeSpecName "kube-api-access-krlgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:00:19.078694 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:19.078660 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krlgp\" (UniqueName: \"kubernetes.io/projected/f6f3275b-d95f-403f-ac0d-207602677186-kube-api-access-krlgp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:00:19.676550 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:19.676515 5086 generic.go:356] "Generic (PLEG): container finished" podID="f6f3275b-d95f-403f-ac0d-207602677186" containerID="f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3" exitCode=137 Nov 03 11:00:19.677014 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:19.676568 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-53c342" Nov 03 11:00:19.677014 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:19.676611 5086 scope.go:122] "RemoveContainer" containerID="f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3" Nov 03 11:00:19.742570 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:19.742517 5086 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-53c342\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="f6f3275b-d95f-403f-ac0d-207602677186" pod="opendatahub/test-curl-7600-53c342" Nov 03 11:00:19.770672 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:19.770462 5086 scope.go:122] "RemoveContainer" containerID="f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3" Nov 03 11:00:19.771794 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:19.771750 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3\": container with ID starting with f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3 not found: ID does not exist" containerID="f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3" Nov 03 11:00:19.772030 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:19.771984 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3"} err="failed to get container status \"f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3\": rpc error: code = NotFound desc = could not find container \"f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3\": container with ID starting with f51ad1c095cc1e8fdad4636f82b303f5930c68950cb567ada706a97adf21c0e3 not found: ID does not exist" Nov 03 11:00:20.171939 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:20.171907 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f3275b-d95f-403f-ac0d-207602677186" path="/var/lib/kubelet/pods/f6f3275b-d95f-403f-ac0d-207602677186/volumes" Nov 03 11:00:20.896042 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:20.896017 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-b41f69" Nov 03 11:00:20.908291 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:20.908263 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5-kube-api-access-999tx\" (UniqueName: \"kubernetes.io/projected/a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5-kube-api-access-999tx\") pod \"a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5\" (UID: \"a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5\") " Nov 03 11:00:20.910353 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:20.910329 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5-kube-api-access-999tx" pod "a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5" (UID: "a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5"). InnerVolumeSpecName "kube-api-access-999tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:00:21.010129 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:21.010044 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-999tx\" (UniqueName: \"kubernetes.io/projected/a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5-kube-api-access-999tx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:00:21.693918 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:21.693881 5086 generic.go:356] "Generic (PLEG): container finished" podID="a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5" containerID="ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0" exitCode=137 Nov 03 11:00:21.694100 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:21.693933 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-b41f69" Nov 03 11:00:21.694100 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:21.693986 5086 scope.go:122] "RemoveContainer" containerID="ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0" Nov 03 11:00:21.705094 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:21.705074 5086 scope.go:122] "RemoveContainer" containerID="ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0" Nov 03 11:00:21.705426 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:21.705396 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0\": container with ID starting with ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0 not found: ID does not exist" containerID="ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0" Nov 03 11:00:21.705528 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:21.705434 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0"} err="failed to get container status \"ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0\": rpc error: code = NotFound desc = could not find container \"ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0\": container with ID starting with ee7618d53a94334f6eb80bf31d0494b015150bf004a304029188862d6a8042c0 not found: ID does not exist" Nov 03 11:00:21.705528 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:21.705478 5086 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-b41f69\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5" pod="opendatahub/test-curl-7600-b41f69" Nov 03 11:00:22.169498 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:22.169461 5086 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-b41f69\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5" pod="opendatahub/test-curl-7600-b41f69" Nov 03 11:00:22.172245 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:22.172215 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5" path="/var/lib/kubelet/pods/a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5/volumes" Nov 03 11:00:24.275469 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:24.275431 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:00:29.755576 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.755539 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj"] Nov 03 11:00:29.756610 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.756576 5086 cpu_manager.go:414] "containerMap: removing container" podUID="a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5" containerName="test-curl-7600-b41f69" Nov 03 11:00:29.756610 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.756603 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5" containerName="test-curl-7600-b41f69" Nov 03 11:00:29.756809 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.756797 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b2d98a-5a92-4ab3-a2fa-5f138e9a47b5" containerName="test-curl-7600-b41f69" Nov 03 11:00:29.760845 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.760816 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.772453 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.772426 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-47a8a4-openshift-default-dockercfg-97wwj\"" Nov 03 11:00:29.774780 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.774752 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj"] Nov 03 11:00:29.863137 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.863099 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-data\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.863316 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.863149 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-token\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.863316 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.863221 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeb4aef-28a7-447c-a5f9-7992a41382f9-istiod-ca-cert\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.863432 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.863352 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-credential-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.863432 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.863397 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-certs\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.863569 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.863551 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-envoy\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.863636 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.863621 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-podinfo\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.863688 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.863648 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.863737 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.863717 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qkx\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-kube-api-access-h5qkx\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.964842 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.964794 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-podinfo\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965043 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.964849 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965043 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.964878 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qkx\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-kube-api-access-h5qkx\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965043 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965018 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-data\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965234 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965051 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-token\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965234 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965086 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeb4aef-28a7-447c-a5f9-7992a41382f9-istiod-ca-cert\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965234 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965134 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-credential-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965234 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965182 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-certs\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965463 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965329 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965463 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965335 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-envoy\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965463 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965427 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-data\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965697 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965675 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-credential-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965805 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965784 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-certs\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.965897 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.965881 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeb4aef-28a7-447c-a5f9-7992a41382f9-istiod-ca-cert\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.967363 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.967342 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-envoy\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.967504 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.967485 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-podinfo\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.972429 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.972408 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qkx\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-kube-api-access-h5qkx\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:29.972524 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:29.972447 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-token\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:30.089656 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:30.089617 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:30.375577 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:30.375031 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj"] Nov 03 11:00:30.375701 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:00:30.375618 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaeb4aef_28a7_447c_a5f9_7992a41382f9.slice/crio-26a8ea6538450b21b823a060d70e2cd3af3a5d73a71e84597a2025aaaef74cbc WatchSource:0}: Error finding container 26a8ea6538450b21b823a060d70e2cd3af3a5d73a71e84597a2025aaaef74cbc: Status 404 returned error can't find the container with id 26a8ea6538450b21b823a060d70e2cd3af3a5d73a71e84597a2025aaaef74cbc Nov 03 11:00:30.378198 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:30.378157 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:00:30.378333 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:30.378228 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:00:30.378333 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:30.378256 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:00:30.783595 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:30.783506 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" event={"ID":"faeb4aef-28a7-447c-a5f9-7992a41382f9","Type":"ContainerStarted","Data":"f3c0dc1645d1e20b8841897eeff95a38af7224090e2d612f25ff0bf83c4feee7"} Nov 03 11:00:30.783595 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:30.783575 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" event={"ID":"faeb4aef-28a7-447c-a5f9-7992a41382f9","Type":"ContainerStarted","Data":"26a8ea6538450b21b823a060d70e2cd3af3a5d73a71e84597a2025aaaef74cbc"} Nov 03 11:00:30.809336 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:30.809263 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" podStartSLOduration=1.809244836 podStartE2EDuration="1.809244836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:30.804221033 +0000 UTC m=+1908.789092316" watchObservedRunningTime="2026-11-03 11:00:30.809244836 +0000 UTC m=+1908.794116120" Nov 03 11:00:31.090178 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:31.090145 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:31.091675 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:31.091641 5086 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 11:00:31.091814 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:31.091705 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 11:00:32.091049 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:32.091008 5086 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 11:00:32.091836 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:32.091169 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 11:00:33.090139 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:33.090101 5086 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 11:00:33.090325 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:33.090162 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 11:00:34.090786 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.090750 5086 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 11:00:34.091174 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.090803 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 11:00:34.456158 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.456063 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm"] Nov 03 11:00:34.460674 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.460653 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.465720 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.465687 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-185908-serving-cert\"" Nov 03 11:00:34.467725 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.467707 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.467960 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.467933 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-metrics-certs\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.468061 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.467993 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.468630 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.468187 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz6pn\" (UniqueName: \"kubernetes.io/projected/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-kube-api-access-wz6pn\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.471687 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.471665 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm"] Nov 03 11:00:34.571513 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.571458 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz6pn\" (UniqueName: \"kubernetes.io/projected/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-kube-api-access-wz6pn\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.572114 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.572080 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.572252 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.572156 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-metrics-certs\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.572252 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.572224 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.572493 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:34.572472 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-serving-cert: secret "maas-api-e2e-extmod-a-185908-serving-cert" not found Nov 03 11:00:34.572679 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:34.572567 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls podName:ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc nodeName:}" failed. No retries permitted until 2026-11-03 11:00:35.072546302 +0000 UTC m=+1913.057417581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls") pod "maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" (UID: "ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc") : secret "maas-api-e2e-extmod-a-185908-serving-cert" not found Nov 03 11:00:34.576485 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.572902 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.577079 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.577045 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-metrics-certs\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:34.580107 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:34.580078 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz6pn\" (UniqueName: \"kubernetes.io/projected/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-kube-api-access-wz6pn\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:35.086255 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:35.086221 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:35.088923 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:35.088895 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:35.090993 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:35.090965 5086 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 11:00:35.091564 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:35.091031 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 11:00:35.378838 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:35.378751 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:35.688926 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:00:35.688898 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8ec9b0_ec37_49ec_97ec_63c0a3ec50cc.slice/crio-969e4acf31d329fa7dc906321d4be52800c3cf047fa9f1bb51554cde89a0a3a7 WatchSource:0}: Error finding container 969e4acf31d329fa7dc906321d4be52800c3cf047fa9f1bb51554cde89a0a3a7: Status 404 returned error can't find the container with id 969e4acf31d329fa7dc906321d4be52800c3cf047fa9f1bb51554cde89a0a3a7 Nov 03 11:00:35.689681 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:35.689654 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm"] Nov 03 11:00:35.845032 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:35.844999 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" event={"ID":"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc","Type":"ContainerStarted","Data":"969e4acf31d329fa7dc906321d4be52800c3cf047fa9f1bb51554cde89a0a3a7"} Nov 03 11:00:36.090429 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:36.090399 5086 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 11:00:36.090614 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:36.090452 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 11:00:36.854760 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:36.854718 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" event={"ID":"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc","Type":"ContainerStarted","Data":"7bfdfe1e5c52f6f9ec96996630485d51854115ef8867be1654e4c374fccef9ac"} Nov 03 11:00:36.855226 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:36.854814 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:36.873097 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:36.873047 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" podStartSLOduration=2.290472475 podStartE2EDuration="2.873030753s" totalImagesPullingTime="582.558278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:36.869355106 +0000 UTC m=+1914.854226388" watchObservedRunningTime="2026-11-03 11:00:36.873030753 +0000 UTC m=+1914.857902037" Nov 03 11:00:37.291395 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:37.291288 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:37.291916 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:37.291880 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:37.292924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:37.292899 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:00:42.868056 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:42.868027 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:00:46.964514 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:46.964376 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v"] Nov 03 11:00:49.107074 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:00:49.107036 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8"] Nov 03 11:00:54.275022 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:00:54.274941 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:01:01.997837 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:01.997769 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" podUID="4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" containerName="main" containerID="cri-o://b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7" gracePeriod=45 Nov 03 11:01:02.285807 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.285781 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 11:01:02.461693 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.461651 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tmp-dir\") pod \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " Nov 03 11:01:02.461900 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.461755 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-dshm\") pod \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " Nov 03 11:01:02.461900 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.461815 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-home\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-home\") pod \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " Nov 03 11:01:02.461900 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.461863 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-model-cache\") pod \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " Nov 03 11:01:02.461900 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.461891 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tls-certs\") pod \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " Nov 03 11:01:02.462107 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.461953 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tmp-dir" pod "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" (UID: "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:01:02.462107 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.461982 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-kube-api-access-kgs9m\" (UniqueName: \"kubernetes.io/projected/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-kube-api-access-kgs9m\") pod \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\" (UID: \"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c\") " Nov 03 11:01:02.462207 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.462133 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-model-cache" pod "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" (UID: "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:01:02.462207 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.462141 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-home" pod "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" (UID: "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:01:02.462572 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.462552 5086 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:02.462668 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.462575 5086 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:02.462668 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.462587 5086 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:02.464123 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.464086 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-dshm" pod "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" (UID: "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:01:02.464233 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.464101 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tls-certs" pod "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" (UID: "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:01:02.464233 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.464189 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-kube-api-access-kgs9m" pod "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" (UID: "4ca4cbcb-5336-40c3-b669-4b47cc7afb1c"). InnerVolumeSpecName "kube-api-access-kgs9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:01:02.564521 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.564446 5086 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:02.564521 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.564474 5086 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:02.564521 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:02.564483 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgs9m\" (UniqueName: \"kubernetes.io/projected/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c-kube-api-access-kgs9m\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:03.087635 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:03.087595 5086 generic.go:356] "Generic (PLEG): container finished" podID="4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" containerID="b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7" exitCode=0 Nov 03 11:01:03.088062 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:03.087640 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" Nov 03 11:01:03.088062 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:03.087673 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" event={"ID":"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c","Type":"ContainerDied","Data":"b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7"} Nov 03 11:01:03.088062 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:03.087719 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v" event={"ID":"4ca4cbcb-5336-40c3-b669-4b47cc7afb1c","Type":"ContainerDied","Data":"08a448502946d33ea0d66565b2703045f49d7c8ece72a138043a674cd9cb4565"} Nov 03 11:01:03.088062 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:03.087736 5086 scope.go:122] "RemoveContainer" containerID="b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7" Nov 03 11:01:03.099244 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:03.099175 5086 scope.go:122] "RemoveContainer" containerID="b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7" Nov 03 11:01:03.099533 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:01:03.099515 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7\": container with ID starting with b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7 not found: ID does not exist" containerID="b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7" Nov 03 11:01:03.099594 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:03.099540 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7"} err="failed to get container status \"b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7\": rpc error: code = NotFound desc = could not find container \"b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7\": container with ID starting with b9d9f6c3a216e454187a2859d5aa1ea580dfcefed427f3695a4af56ced9145e7 not found: ID does not exist" Nov 03 11:01:03.114626 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:03.114597 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v"] Nov 03 11:01:03.118153 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:03.118124 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e92a08/auth-test-model-e92a08-kserve-6cccd9cf85-hhd8v"] Nov 03 11:01:04.145831 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.145793 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" podUID="66c1a91d-c562-47ab-8919-6db7cb914802" containerName="main" containerID="cri-o://d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55" gracePeriod=45 Nov 03 11:01:04.173452 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.173426 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" path="/var/lib/kubelet/pods/4ca4cbcb-5336-40c3-b669-4b47cc7afb1c/volumes" Nov 03 11:01:04.466227 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.466196 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 11:01:04.514178 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.514137 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c1a91d-c562-47ab-8919-6db7cb914802-tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1a91d-c562-47ab-8919-6db7cb914802-tls-certs\") pod \"66c1a91d-c562-47ab-8919-6db7cb914802\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " Nov 03 11:01:04.514385 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.514194 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-dshm\") pod \"66c1a91d-c562-47ab-8919-6db7cb914802\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " Nov 03 11:01:04.514385 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.514324 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-home\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-home\") pod \"66c1a91d-c562-47ab-8919-6db7cb914802\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " Nov 03 11:01:04.514508 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.514454 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-model-cache\") pod \"66c1a91d-c562-47ab-8919-6db7cb914802\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " Nov 03 11:01:04.514603 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.514574 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-home" pod "66c1a91d-c562-47ab-8919-6db7cb914802" (UID: "66c1a91d-c562-47ab-8919-6db7cb914802"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:01:04.514746 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.514682 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-model-cache" pod "66c1a91d-c562-47ab-8919-6db7cb914802" (UID: "66c1a91d-c562-47ab-8919-6db7cb914802"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:01:04.514746 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.514722 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-tmp-dir\") pod \"66c1a91d-c562-47ab-8919-6db7cb914802\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " Nov 03 11:01:04.514860 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.514783 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c1a91d-c562-47ab-8919-6db7cb914802-kube-api-access-v7cnt\" (UniqueName: \"kubernetes.io/projected/66c1a91d-c562-47ab-8919-6db7cb914802-kube-api-access-v7cnt\") pod \"66c1a91d-c562-47ab-8919-6db7cb914802\" (UID: \"66c1a91d-c562-47ab-8919-6db7cb914802\") " Nov 03 11:01:04.514949 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.514926 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-tmp-dir" pod "66c1a91d-c562-47ab-8919-6db7cb914802" (UID: "66c1a91d-c562-47ab-8919-6db7cb914802"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:01:04.515506 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.515484 5086 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:04.515608 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.515509 5086 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:04.515608 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.515526 5086 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:04.516712 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.516685 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c1a91d-c562-47ab-8919-6db7cb914802-tls-certs" pod "66c1a91d-c562-47ab-8919-6db7cb914802" (UID: "66c1a91d-c562-47ab-8919-6db7cb914802"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:01:04.516791 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.516721 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-dshm" pod "66c1a91d-c562-47ab-8919-6db7cb914802" (UID: "66c1a91d-c562-47ab-8919-6db7cb914802"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:01:04.516862 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.516842 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c1a91d-c562-47ab-8919-6db7cb914802-kube-api-access-v7cnt" pod "66c1a91d-c562-47ab-8919-6db7cb914802" (UID: "66c1a91d-c562-47ab-8919-6db7cb914802"). InnerVolumeSpecName "kube-api-access-v7cnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:01:04.616839 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.616802 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7cnt\" (UniqueName: \"kubernetes.io/projected/66c1a91d-c562-47ab-8919-6db7cb914802-kube-api-access-v7cnt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:04.616839 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.616837 5086 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1a91d-c562-47ab-8919-6db7cb914802-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:04.616839 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:04.616850 5086 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1a91d-c562-47ab-8919-6db7cb914802-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:01:05.105469 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:05.105434 5086 generic.go:356] "Generic (PLEG): container finished" podID="66c1a91d-c562-47ab-8919-6db7cb914802" containerID="d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55" exitCode=0 Nov 03 11:01:05.105665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:05.105479 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" Nov 03 11:01:05.105665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:05.105484 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" event={"ID":"66c1a91d-c562-47ab-8919-6db7cb914802","Type":"ContainerDied","Data":"d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55"} Nov 03 11:01:05.105665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:05.105584 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8" event={"ID":"66c1a91d-c562-47ab-8919-6db7cb914802","Type":"ContainerDied","Data":"40826f71c166118e810ff13ac1390e20a7086dfaf20265ae89131f8d8a225ea1"} Nov 03 11:01:05.105665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:05.105603 5086 scope.go:122] "RemoveContainer" containerID="d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55" Nov 03 11:01:05.116380 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:05.116361 5086 scope.go:122] "RemoveContainer" containerID="d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55" Nov 03 11:01:05.116637 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:01:05.116616 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55\": container with ID starting with d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55 not found: ID does not exist" containerID="d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55" Nov 03 11:01:05.116750 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:05.116646 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55"} err="failed to get container status \"d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55\": rpc error: code = NotFound desc = could not find container \"d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55\": container with ID starting with d0afeb13a7671ce45eaca242baa4f5242a2c79441093c09e988b7d9929c55a55 not found: ID does not exist" Nov 03 11:01:05.128721 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:05.128697 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8"] Nov 03 11:01:05.133763 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:05.133742 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-fcc2ce/auth-test-model-fcc2ce-kserve-865d468cbf-pkml8"] Nov 03 11:01:06.176973 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.176938 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c1a91d-c562-47ab-8919-6db7cb914802" path="/var/lib/kubelet/pods/66c1a91d-c562-47ab-8919-6db7cb914802/volumes" Nov 03 11:01:06.760991 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.760957 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9"] Nov 03 11:01:06.762072 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.762046 5086 cpu_manager.go:414] "containerMap: removing container" podUID="4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" containerName="main" Nov 03 11:01:06.762185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.762074 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" containerName="main" Nov 03 11:01:06.762185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.762143 5086 cpu_manager.go:414] "containerMap: removing container" podUID="66c1a91d-c562-47ab-8919-6db7cb914802" containerName="main" Nov 03 11:01:06.762185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.762152 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c1a91d-c562-47ab-8919-6db7cb914802" containerName="main" Nov 03 11:01:06.762490 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.762371 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c1a91d-c562-47ab-8919-6db7cb914802" containerName="main" Nov 03 11:01:06.762490 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.762397 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca4cbcb-5336-40c3-b669-4b47cc7afb1c" containerName="main" Nov 03 11:01:06.785562 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.785528 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9"] Nov 03 11:01:06.785746 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.785645 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.810540 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.810506 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-917da2-serving-cert\"" Nov 03 11:01:06.852414 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.852384 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.852559 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.852430 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-maas-api-tls\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.852559 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.852502 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-metrics-certs\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.852694 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.852656 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxs8r\" (UniqueName: \"kubernetes.io/projected/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-kube-api-access-mxs8r\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.954675 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.954638 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.954865 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.954687 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-maas-api-tls\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.954865 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.954710 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-metrics-certs\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.954865 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.954767 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxs8r\" (UniqueName: \"kubernetes.io/projected/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-kube-api-access-mxs8r\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.955550 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.955520 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.957838 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.957811 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-maas-api-tls\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.957977 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.957811 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-metrics-certs\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:06.964703 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:06.964677 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxs8r\" (UniqueName: \"kubernetes.io/projected/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-kube-api-access-mxs8r\") pod \"maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:07.105517 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:07.105483 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:07.429251 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:07.429223 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9"] Nov 03 11:01:08.228788 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:08.227488 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" event={"ID":"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91","Type":"ContainerStarted","Data":"9021f2fb00782ed847a1596a9e1a08e5c44fa74362f7c98257fbf19277b47c9f"} Nov 03 11:01:08.228788 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:08.227574 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" event={"ID":"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91","Type":"ContainerStarted","Data":"25e4add8040e834637373038f0d7b61d20e53f3386051c3d3dbb44c3cc5728d7"} Nov 03 11:01:08.228788 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:08.228679 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:08.249980 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:08.249928 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" podStartSLOduration=1.763821354 podStartE2EDuration="2.249913063s" totalImagesPullingTime="486.091709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:08.248815625 +0000 UTC m=+1946.233686934" watchObservedRunningTime="2026-11-03 11:01:08.249913063 +0000 UTC m=+1946.234784345" Nov 03 11:01:09.336428 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.336392 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b"] Nov 03 11:01:09.343509 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.342424 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.349668 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.349638 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b"] Nov 03 11:01:09.349916 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.349898 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-cbeebd-serving-cert\"" Nov 03 11:01:09.521321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.520750 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.521321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.520810 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1852843-24f8-4260-acb4-e4c38514a646-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.521321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.520943 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wst\" (UniqueName: \"kubernetes.io/projected/d1852843-24f8-4260-acb4-e4c38514a646-kube-api-access-h2wst\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.521321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.520994 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-metrics-certs\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.622902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.622807 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wst\" (UniqueName: \"kubernetes.io/projected/d1852843-24f8-4260-acb4-e4c38514a646-kube-api-access-h2wst\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.623618 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.623591 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-metrics-certs\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.624647 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:01:09.624630 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-serving-cert: secret "maas-api-e2e-extmod-a-cbeebd-serving-cert" not found Nov 03 11:01:09.625032 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.625013 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.625211 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:01:09.625190 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls podName:d1852843-24f8-4260-acb4-e4c38514a646 nodeName:}" failed. No retries permitted until 2026-11-03 11:01:10.125163423 +0000 UTC m=+1948.110034691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls") pod "maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" (UID: "d1852843-24f8-4260-acb4-e4c38514a646") : secret "maas-api-e2e-extmod-a-cbeebd-serving-cert" not found Nov 03 11:01:09.625378 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.625359 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1852843-24f8-4260-acb4-e4c38514a646-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.653315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.630843 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-metrics-certs\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.653315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.641722 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1852843-24f8-4260-acb4-e4c38514a646-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:09.659788 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:09.656914 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wst\" (UniqueName: \"kubernetes.io/projected/d1852843-24f8-4260-acb4-e4c38514a646-kube-api-access-h2wst\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:10.155321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:10.154636 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:10.174544 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:10.171097 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:10.271725 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:10.271695 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:10.592390 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:10.592351 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b"] Nov 03 11:01:10.592700 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:01:10.592541 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1852843_24f8_4260_acb4_e4c38514a646.slice/crio-e4f2cc302de218c018395c14f3ae34414428bcf05f46704a7c2f3f4c83300f64 WatchSource:0}: Error finding container e4f2cc302de218c018395c14f3ae34414428bcf05f46704a7c2f3f4c83300f64: Status 404 returned error can't find the container with id e4f2cc302de218c018395c14f3ae34414428bcf05f46704a7c2f3f4c83300f64 Nov 03 11:01:10.594034 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:10.594016 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 11:01:11.252974 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:11.252937 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" event={"ID":"d1852843-24f8-4260-acb4-e4c38514a646","Type":"ContainerStarted","Data":"a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0"} Nov 03 11:01:11.253146 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:11.252996 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" event={"ID":"d1852843-24f8-4260-acb4-e4c38514a646","Type":"ContainerStarted","Data":"e4f2cc302de218c018395c14f3ae34414428bcf05f46704a7c2f3f4c83300f64"} Nov 03 11:01:11.253146 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:11.253021 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:11.270829 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:11.270777 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" podStartSLOduration=1.837155321 podStartE2EDuration="2.270763293s" totalImagesPullingTime="433.607972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:11.268141483 +0000 UTC m=+1949.253012768" watchObservedRunningTime="2026-11-03 11:01:11.270763293 +0000 UTC m=+1949.255634575" Nov 03 11:01:15.251168 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:15.251135 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:01:17.264559 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:17.264529 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:01:19.972032 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:19.971998 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g"] Nov 03 11:01:19.981470 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:19.981430 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g"] Nov 03 11:01:19.981611 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:19.981565 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:19.990059 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:19.990034 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-56687c-serving-cert\"" Nov 03 11:01:20.120898 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.120857 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-metrics-certs\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.123955 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.123918 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a443b-f6da-4644-a6ae-198e22602468-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.128266 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.128234 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5xz\" (UniqueName: \"kubernetes.io/projected/482a443b-f6da-4644-a6ae-198e22602468-kube-api-access-hn5xz\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.128665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.128617 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-maas-api-tls\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.229743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.229652 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a443b-f6da-4644-a6ae-198e22602468-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.229924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.229772 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5xz\" (UniqueName: \"kubernetes.io/projected/482a443b-f6da-4644-a6ae-198e22602468-kube-api-access-hn5xz\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.229924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.229867 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-maas-api-tls\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.230045 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.229985 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-metrics-certs\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.230428 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.230408 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a443b-f6da-4644-a6ae-198e22602468-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.232658 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.232627 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-maas-api-tls\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.232769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.232659 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-metrics-certs\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.237474 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.237452 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5xz\" (UniqueName: \"kubernetes.io/projected/482a443b-f6da-4644-a6ae-198e22602468-kube-api-access-hn5xz\") pod \"maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.300169 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.300138 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:20.584704 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:20.584671 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g"] Nov 03 11:01:20.585612 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:01:20.585588 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482a443b_f6da_4644_a6ae_198e22602468.slice/crio-331d7966ceea7cbc8bf465b99a497671aa98c9f42d125ef7ea01375d826dcf2f WatchSource:0}: Error finding container 331d7966ceea7cbc8bf465b99a497671aa98c9f42d125ef7ea01375d826dcf2f: Status 404 returned error can't find the container with id 331d7966ceea7cbc8bf465b99a497671aa98c9f42d125ef7ea01375d826dcf2f Nov 03 11:01:21.347211 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:21.347180 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" event={"ID":"482a443b-f6da-4644-a6ae-198e22602468","Type":"ContainerStarted","Data":"1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a"} Nov 03 11:01:21.347725 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:21.347239 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" event={"ID":"482a443b-f6da-4644-a6ae-198e22602468","Type":"ContainerStarted","Data":"331d7966ceea7cbc8bf465b99a497671aa98c9f42d125ef7ea01375d826dcf2f"} Nov 03 11:01:21.347725 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:21.347270 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:21.367009 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:21.366960 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" podStartSLOduration=1.93039129 podStartE2EDuration="2.366945611s" totalImagesPullingTime="436.554321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:21.3632606 +0000 UTC m=+1959.348131888" watchObservedRunningTime="2026-11-03 11:01:21.366945611 +0000 UTC m=+1959.351816935" Nov 03 11:01:24.304308 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:01:24.304172 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:01:27.360631 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:27.360596 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:01:47.409854 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.409818 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph"] Nov 03 11:01:47.435245 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.435204 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph"] Nov 03 11:01:47.435449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.435388 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.445737 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.445713 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-fb5766-serving-cert\"" Nov 03 11:01:47.587102 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.587056 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-metrics-certs\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.587545 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.587522 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.587700 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.587683 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td7pf\" (UniqueName: \"kubernetes.io/projected/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-kube-api-access-td7pf\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.587846 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.587833 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.689647 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.689555 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-metrics-certs\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.689827 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.689690 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.689827 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.689716 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td7pf\" (UniqueName: \"kubernetes.io/projected/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-kube-api-access-td7pf\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.689827 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.689763 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.690002 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:01:47.689916 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-serving-cert: secret "maas-api-e2e-extmod-b-fb5766-serving-cert" not found Nov 03 11:01:47.690002 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:01:47.689988 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls podName:369e9ba2-4a9d-4a48-9fa3-7a10ad04803a nodeName:}" failed. No retries permitted until 2026-11-03 11:01:48.189965826 +0000 UTC m=+1986.174837102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls") pod "maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" (UID: "369e9ba2-4a9d-4a48-9fa3-7a10ad04803a") : secret "maas-api-e2e-extmod-b-fb5766-serving-cert" not found Nov 03 11:01:47.690482 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.690463 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.692419 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.692395 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-metrics-certs\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:47.697961 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:47.697939 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td7pf\" (UniqueName: \"kubernetes.io/projected/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-kube-api-access-td7pf\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:48.201669 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:48.201629 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:48.205026 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:48.204993 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:48.368876 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:48.368843 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:48.664425 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:01:48.664392 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369e9ba2_4a9d_4a48_9fa3_7a10ad04803a.slice/crio-faf32b9055adbbb7a213909fcd92fca0d107579bf96eb7130e947e156b1e55da WatchSource:0}: Error finding container faf32b9055adbbb7a213909fcd92fca0d107579bf96eb7130e947e156b1e55da: Status 404 returned error can't find the container with id faf32b9055adbbb7a213909fcd92fca0d107579bf96eb7130e947e156b1e55da Nov 03 11:01:48.665578 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:48.665551 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph"] Nov 03 11:01:49.605175 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:49.605136 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" event={"ID":"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a","Type":"ContainerStarted","Data":"da86201f51df02ea5401dcc82e9acf46a2f98ac18b49e40b3cfb1750a4721ff3"} Nov 03 11:01:49.605379 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:49.605196 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" event={"ID":"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a","Type":"ContainerStarted","Data":"faf32b9055adbbb7a213909fcd92fca0d107579bf96eb7130e947e156b1e55da"} Nov 03 11:01:49.605379 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:49.605255 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:01:49.622563 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:49.622513 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" podStartSLOduration=2.128987329 podStartE2EDuration="2.622498984s" totalImagesPullingTime="493.511655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:49.620051466 +0000 UTC m=+1987.604922762" watchObservedRunningTime="2026-11-03 11:01:49.622498984 +0000 UTC m=+1987.607370281" Nov 03 11:01:54.291816 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:01:54.291778 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:01:55.618738 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:01:55.618710 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:02:00.386002 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.385966 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7"] Nov 03 11:02:00.392043 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.392016 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.406436 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.398739 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b76c62-serving-cert\"" Nov 03 11:02:00.406436 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.399902 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7"] Nov 03 11:02:00.500054 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.500022 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.500238 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.500062 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee98b2af-6d15-403a-99b6-02aeb95b98af-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.500238 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.500089 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-metrics-certs\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.500238 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.500195 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxhm\" (UniqueName: \"kubernetes.io/projected/ee98b2af-6d15-403a-99b6-02aeb95b98af-kube-api-access-mzxhm\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.601603 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.601555 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.601831 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.601628 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee98b2af-6d15-403a-99b6-02aeb95b98af-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.601831 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.601672 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-metrics-certs\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.601831 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.601707 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxhm\" (UniqueName: \"kubernetes.io/projected/ee98b2af-6d15-403a-99b6-02aeb95b98af-kube-api-access-mzxhm\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.602932 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.602901 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee98b2af-6d15-403a-99b6-02aeb95b98af-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.605255 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.605231 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.605687 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.605661 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-metrics-certs\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.610205 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.610174 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxhm\" (UniqueName: \"kubernetes.io/projected/ee98b2af-6d15-403a-99b6-02aeb95b98af-kube-api-access-mzxhm\") pod \"maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:00.725242 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:00.725156 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:01.005607 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:01.005576 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7"] Nov 03 11:02:01.006185 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:02:01.006159 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee98b2af_6d15_403a_99b6_02aeb95b98af.slice/crio-caf8bcdb582bf7f761867ac364a83230991a84047ed5c43220c9c4b87843cb4e WatchSource:0}: Error finding container caf8bcdb582bf7f761867ac364a83230991a84047ed5c43220c9c4b87843cb4e: Status 404 returned error can't find the container with id caf8bcdb582bf7f761867ac364a83230991a84047ed5c43220c9c4b87843cb4e Nov 03 11:02:01.692697 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:01.692661 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" event={"ID":"ee98b2af-6d15-403a-99b6-02aeb95b98af","Type":"ContainerStarted","Data":"b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4"} Nov 03 11:02:01.693087 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:01.692723 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" event={"ID":"ee98b2af-6d15-403a-99b6-02aeb95b98af","Type":"ContainerStarted","Data":"caf8bcdb582bf7f761867ac364a83230991a84047ed5c43220c9c4b87843cb4e"} Nov 03 11:02:01.693087 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:01.692763 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:01.711404 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:01.711355 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" podStartSLOduration=1.24622614 podStartE2EDuration="1.711340099s" totalImagesPullingTime="465.113959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:02:01.708919401 +0000 UTC m=+1999.693790685" watchObservedRunningTime="2026-11-03 11:02:01.711340099 +0000 UTC m=+1999.696211384" Nov 03 11:02:07.707668 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:07.707639 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:02:16.288813 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:16.288774 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq"] Nov 03 11:02:16.289201 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:16.289000 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" containerID="cri-o://d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400" gracePeriod=30 Nov 03 11:02:21.588973 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.588910 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 11:02:21.666038 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.666001 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-podinfo\") pod \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " Nov 03 11:02:21.666239 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.666059 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-credential-socket\") pod \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " Nov 03 11:02:21.666239 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.666133 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-token\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-token\") pod \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " Nov 03 11:02:21.666239 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.666169 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-kube-api-access-kqx55\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-kube-api-access-kqx55\") pod \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " Nov 03 11:02:21.666239 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.666205 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-socket\") pod \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " Nov 03 11:02:21.666525 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.666340 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istiod-ca-cert\") pod \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " Nov 03 11:02:21.666525 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.666447 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-envoy\") pod \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " Nov 03 11:02:21.666652 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.666567 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-data\") pod \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " Nov 03 11:02:21.666652 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.666637 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-certs\") pod \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\" (UID: \"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56\") " Nov 03 11:02:21.667751 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.667720 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-certs" pod "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" (UID: "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:21.668283 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.668258 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-socket" pod "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" (UID: "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:21.668665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.668639 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-credential-socket" pod "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" (UID: "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:21.668914 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.668887 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-data" pod "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" (UID: "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:21.669070 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.669046 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istiod-ca-cert" pod "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" (UID: "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:02:21.672704 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.672669 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-envoy" pod "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" (UID: "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:21.673731 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.673698 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-podinfo" pod "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" (UID: "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:02:21.673848 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.673803 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-kube-api-access-kqx55" pod "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" (UID: "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56"). InnerVolumeSpecName "kube-api-access-kqx55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:02:21.673979 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.673958 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-token" pod "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" (UID: "72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:02:21.769195 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.769162 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:21.769195 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.769192 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:21.769195 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.769203 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:21.769552 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.769213 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:21.769552 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.769221 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:21.769552 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.769231 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:21.769552 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.769239 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:21.769552 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.769248 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqx55\" (UniqueName: \"kubernetes.io/projected/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-kube-api-access-kqx55\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:21.769552 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.769261 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:21.893528 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.893438 5086 generic.go:356] "Generic (PLEG): container finished" podID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerID="d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400" exitCode=0 Nov 03 11:02:21.893679 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.893547 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" Nov 03 11:02:21.893679 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.893579 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" event={"ID":"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56","Type":"ContainerDied","Data":"d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400"} Nov 03 11:02:21.893679 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.893636 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq" event={"ID":"72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56","Type":"ContainerDied","Data":"50c73dd88d381c9ddda4c05fdc255ac13fe8d435d26c065b7aaf22d4c6ee67cd"} Nov 03 11:02:21.893679 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.893665 5086 scope.go:122] "RemoveContainer" containerID="d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400" Nov 03 11:02:21.926247 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.926219 5086 scope.go:122] "RemoveContainer" containerID="d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400" Nov 03 11:02:21.927206 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:02:21.927181 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400\": container with ID starting with d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400 not found: ID does not exist" containerID="d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400" Nov 03 11:02:21.927336 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.927213 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400"} err="failed to get container status \"d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400\": rpc error: code = NotFound desc = could not find container \"d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400\": container with ID starting with d745aaec80ac77fb98713d9c694011099a996665753ece84efc75bcef08a2400 not found: ID does not exist" Nov 03 11:02:21.940417 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.940387 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq"] Nov 03 11:02:21.943864 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:21.943835 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-88ppq"] Nov 03 11:02:22.173531 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:22.173460 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" path="/var/lib/kubelet/pods/72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56/volumes" Nov 03 11:02:24.397944 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:02:24.397850 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:02:41.479621 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:41.479584 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77"] Nov 03 11:02:41.480141 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:41.479841 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" containerID="cri-o://0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80" gracePeriod=30 Nov 03 11:02:44.121759 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:44.121727 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95"] Nov 03 11:02:44.122160 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:44.121905 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" containerID="cri-o://faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9" gracePeriod=30 Nov 03 11:02:46.780988 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.780954 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 11:02:46.901670 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.901635 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-podinfo\") pod \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " Nov 03 11:02:46.901872 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.901682 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-token\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-token\") pod \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " Nov 03 11:02:46.901872 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.901711 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-data\") pod \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " Nov 03 11:02:46.901872 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.901749 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-credential-socket\") pod \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " Nov 03 11:02:46.901872 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.901784 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-certs\") pod \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " Nov 03 11:02:46.901872 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.901842 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-kube-api-access-cccpp\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-kube-api-access-cccpp\") pod \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " Nov 03 11:02:46.902162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.901950 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-envoy\") pod \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " Nov 03 11:02:46.902162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.902024 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-credential-socket" pod "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" (UID: "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:46.902162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.902042 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istiod-ca-cert\") pod \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " Nov 03 11:02:46.902162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.902056 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-data" pod "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" (UID: "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:46.902162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.902099 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-certs" pod "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" (UID: "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:46.902489 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.902183 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-socket\") pod \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\" (UID: \"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4\") " Nov 03 11:02:46.902489 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.902440 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istiod-ca-cert" pod "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" (UID: "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:02:46.902819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.902795 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-socket" pod "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" (UID: "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:46.903106 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.903087 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:46.903210 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.903114 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:46.903210 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.903129 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:46.903210 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.903145 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:46.903210 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.903170 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:46.904382 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.904353 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-token" pod "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" (UID: "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:02:46.904497 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.904455 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-envoy" pod "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" (UID: "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:46.904593 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.904567 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-podinfo" pod "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" (UID: "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:02:46.904647 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:46.904580 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-kube-api-access-cccpp" pod "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" (UID: "5f7a978d-5cfe-4b97-94ea-6b825fa2eac4"). InnerVolumeSpecName "kube-api-access-cccpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:02:47.005241 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.005207 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:47.005241 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.005235 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:47.005241 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.005245 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cccpp\" (UniqueName: \"kubernetes.io/projected/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-kube-api-access-cccpp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:47.005508 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.005256 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:47.153216 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.153181 5086 generic.go:356] "Generic (PLEG): container finished" podID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerID="0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80" exitCode=0 Nov 03 11:02:47.153427 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.153232 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" Nov 03 11:02:47.153427 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.153272 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" event={"ID":"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4","Type":"ContainerDied","Data":"0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80"} Nov 03 11:02:47.153427 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.153380 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77" event={"ID":"5f7a978d-5cfe-4b97-94ea-6b825fa2eac4","Type":"ContainerDied","Data":"8931d6eef0525a94fb5c3c9eb4327755d415b89c891fe7f889909fe49b2c839f"} Nov 03 11:02:47.153427 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.153408 5086 scope.go:122] "RemoveContainer" containerID="0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80" Nov 03 11:02:47.164859 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.164844 5086 scope.go:122] "RemoveContainer" containerID="0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80" Nov 03 11:02:47.165132 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:02:47.165111 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80\": container with ID starting with 0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80 not found: ID does not exist" containerID="0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80" Nov 03 11:02:47.165198 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.165142 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80"} err="failed to get container status \"0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80\": rpc error: code = NotFound desc = could not find container \"0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80\": container with ID starting with 0c28b708385c4fc0aed12638a616ea2a0476487e4430aef932e0cb7a81c3fb80 not found: ID does not exist" Nov 03 11:02:47.179343 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.179319 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77"] Nov 03 11:02:47.188942 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:47.188918 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tfb77"] Nov 03 11:02:48.176165 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:48.176133 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" path="/var/lib/kubelet/pods/5f7a978d-5cfe-4b97-94ea-6b825fa2eac4/volumes" Nov 03 11:02:49.419326 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.419286 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 11:02:49.463021 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.462991 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c586beae-5009-458b-8704-e8bc2f21f2a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c586beae-5009-458b-8704-e8bc2f21f2a2-istiod-ca-cert\") pod \"c586beae-5009-458b-8704-e8bc2f21f2a2\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " Nov 03 11:02:49.463183 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463078 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-kube-api-access-fkpv7\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-kube-api-access-fkpv7\") pod \"c586beae-5009-458b-8704-e8bc2f21f2a2\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " Nov 03 11:02:49.463249 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463218 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-socket\") pod \"c586beae-5009-458b-8704-e8bc2f21f2a2\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " Nov 03 11:02:49.463329 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463253 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-token\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-token\") pod \"c586beae-5009-458b-8704-e8bc2f21f2a2\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " Nov 03 11:02:49.463329 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463322 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-data\") pod \"c586beae-5009-458b-8704-e8bc2f21f2a2\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " Nov 03 11:02:49.463452 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463369 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-envoy\") pod \"c586beae-5009-458b-8704-e8bc2f21f2a2\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " Nov 03 11:02:49.463452 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463395 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-podinfo\") pod \"c586beae-5009-458b-8704-e8bc2f21f2a2\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " Nov 03 11:02:49.463452 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463433 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c586beae-5009-458b-8704-e8bc2f21f2a2-istiod-ca-cert" pod "c586beae-5009-458b-8704-e8bc2f21f2a2" (UID: "c586beae-5009-458b-8704-e8bc2f21f2a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:02:49.463607 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463479 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-credential-socket\") pod \"c586beae-5009-458b-8704-e8bc2f21f2a2\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " Nov 03 11:02:49.463607 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463545 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-certs\") pod \"c586beae-5009-458b-8704-e8bc2f21f2a2\" (UID: \"c586beae-5009-458b-8704-e8bc2f21f2a2\") " Nov 03 11:02:49.463942 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463746 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-socket" pod "c586beae-5009-458b-8704-e8bc2f21f2a2" (UID: "c586beae-5009-458b-8704-e8bc2f21f2a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:49.464044 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463968 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-certs" pod "c586beae-5009-458b-8704-e8bc2f21f2a2" (UID: "c586beae-5009-458b-8704-e8bc2f21f2a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:49.464044 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.463984 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-data" pod "c586beae-5009-458b-8704-e8bc2f21f2a2" (UID: "c586beae-5009-458b-8704-e8bc2f21f2a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:49.464288 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.464205 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-credential-socket" pod "c586beae-5009-458b-8704-e8bc2f21f2a2" (UID: "c586beae-5009-458b-8704-e8bc2f21f2a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:49.464388 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.464317 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:49.464388 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.464341 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c586beae-5009-458b-8704-e8bc2f21f2a2-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:49.464388 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.464356 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:49.464388 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.464370 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:49.466276 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.466229 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-kube-api-access-fkpv7" pod "c586beae-5009-458b-8704-e8bc2f21f2a2" (UID: "c586beae-5009-458b-8704-e8bc2f21f2a2"). InnerVolumeSpecName "kube-api-access-fkpv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:02:49.466772 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.466673 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-podinfo" pod "c586beae-5009-458b-8704-e8bc2f21f2a2" (UID: "c586beae-5009-458b-8704-e8bc2f21f2a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:02:49.466772 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.466737 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-envoy" pod "c586beae-5009-458b-8704-e8bc2f21f2a2" (UID: "c586beae-5009-458b-8704-e8bc2f21f2a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:02:49.467255 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.467065 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-token" pod "c586beae-5009-458b-8704-e8bc2f21f2a2" (UID: "c586beae-5009-458b-8704-e8bc2f21f2a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:02:49.565619 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.565587 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:49.565619 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.565616 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:49.565822 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.565628 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c586beae-5009-458b-8704-e8bc2f21f2a2-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:49.565822 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.565638 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkpv7\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-kube-api-access-fkpv7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:49.565822 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:49.565648 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c586beae-5009-458b-8704-e8bc2f21f2a2-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:02:50.178529 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:50.178493 5086 generic.go:356] "Generic (PLEG): container finished" podID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerID="faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9" exitCode=0 Nov 03 11:02:50.178743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:50.178539 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" Nov 03 11:02:50.178743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:50.178568 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" event={"ID":"c586beae-5009-458b-8704-e8bc2f21f2a2","Type":"ContainerDied","Data":"faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9"} Nov 03 11:02:50.178743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:50.178624 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95" event={"ID":"c586beae-5009-458b-8704-e8bc2f21f2a2","Type":"ContainerDied","Data":"5b606b25db98034fde034c2327f778cfad585c155379c34bfbc78454edae3fe1"} Nov 03 11:02:50.178743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:50.178656 5086 scope.go:122] "RemoveContainer" containerID="faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9" Nov 03 11:02:50.192635 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:50.192601 5086 scope.go:122] "RemoveContainer" containerID="faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9" Nov 03 11:02:50.193765 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:02:50.193737 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9\": container with ID starting with faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9 not found: ID does not exist" containerID="faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9" Nov 03 11:02:50.193881 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:50.193777 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9"} err="failed to get container status \"faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9\": rpc error: code = NotFound desc = could not find container \"faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9\": container with ID starting with faf128ca5b903129d3b015349357019f3dc118d1c485b3efb3deb2f2710795a9 not found: ID does not exist" Nov 03 11:02:50.203471 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:50.203445 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95"] Nov 03 11:02:50.206939 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:50.206915 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-s9l95"] Nov 03 11:02:52.171759 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:02:52.171725 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" path="/var/lib/kubelet/pods/c586beae-5009-458b-8704-e8bc2f21f2a2/volumes" Nov 03 11:02:54.276862 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:02:54.276807 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:03:24.276521 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:03:24.276479 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:03:42.984841 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:42.984810 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:03:42.991010 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:42.990974 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:03:43.008940 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:43.008904 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:03:43.014421 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:43.014395 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:03:53.667654 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.667537 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt"] Nov 03 11:03:53.668688 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.668653 5086 cpu_manager.go:414] "containerMap: removing container" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" Nov 03 11:03:53.668688 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.668676 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" Nov 03 11:03:53.668885 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.668702 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" Nov 03 11:03:53.668885 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.668710 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" Nov 03 11:03:53.668885 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.668758 5086 cpu_manager.go:414] "containerMap: removing container" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" Nov 03 11:03:53.668885 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.668766 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" Nov 03 11:03:53.669055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.668928 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7a978d-5cfe-4b97-94ea-6b825fa2eac4" containerName="istio-proxy" Nov 03 11:03:53.669055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.668939 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c586beae-5009-458b-8704-e8bc2f21f2a2" containerName="istio-proxy" Nov 03 11:03:53.669055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.668958 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a7f5f3-2a0f-4f34-aaa1-8dc03b2dbc56" containerName="istio-proxy" Nov 03 11:03:53.675459 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.675429 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.682394 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.682364 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt"] Nov 03 11:03:53.690083 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.690053 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d9f2ce-serving-cert\"" Nov 03 11:03:53.757670 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.757622 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14889afa-093e-40af-93d5-28a889c72d6c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.757926 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.757906 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rz57\" (UniqueName: \"kubernetes.io/projected/14889afa-093e-40af-93d5-28a889c72d6c-kube-api-access-9rz57\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.758005 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.757970 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-metrics-certs\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.758065 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.758012 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.859114 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.859078 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rz57\" (UniqueName: \"kubernetes.io/projected/14889afa-093e-40af-93d5-28a889c72d6c-kube-api-access-9rz57\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.859290 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.859137 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-metrics-certs\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.859290 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.859173 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.859290 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.859252 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14889afa-093e-40af-93d5-28a889c72d6c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.860053 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.860020 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14889afa-093e-40af-93d5-28a889c72d6c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.862069 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.862041 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-metrics-certs\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.862968 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.862656 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:53.895064 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:53.895026 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rz57\" (UniqueName: \"kubernetes.io/projected/14889afa-093e-40af-93d5-28a889c72d6c-kube-api-access-9rz57\") pod \"maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:54.017172 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:54.017087 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:54.320498 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:03:54.320441 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:03:54.326836 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:54.326803 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt"] Nov 03 11:03:54.328578 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:03:54.328548 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14889afa_093e_40af_93d5_28a889c72d6c.slice/crio-b09d2e8618599cfbccaf4d782bf9e06986d34de846d2a55c0e39e004bd0c6c38 WatchSource:0}: Error finding container b09d2e8618599cfbccaf4d782bf9e06986d34de846d2a55c0e39e004bd0c6c38: Status 404 returned error can't find the container with id b09d2e8618599cfbccaf4d782bf9e06986d34de846d2a55c0e39e004bd0c6c38 Nov 03 11:03:54.731381 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:54.731268 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" event={"ID":"14889afa-093e-40af-93d5-28a889c72d6c","Type":"ContainerStarted","Data":"b09d2e8618599cfbccaf4d782bf9e06986d34de846d2a55c0e39e004bd0c6c38"} Nov 03 11:03:55.740281 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:55.740238 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" event={"ID":"14889afa-093e-40af-93d5-28a889c72d6c","Type":"ContainerStarted","Data":"f8741e04c611319868cc010259d60f3319e13d6aa0cc0151e8bb0e4d84ea4229"} Nov 03 11:03:55.740708 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:55.740333 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:03:55.763152 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:03:55.763092 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" podStartSLOduration=2.231114326 podStartE2EDuration="2.763073387s" totalImagesPullingTime="531.959061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:55.758287392 +0000 UTC m=+2113.743158674" watchObservedRunningTime="2026-11-03 11:03:55.763073387 +0000 UTC m=+2113.747944672" Nov 03 11:04:01.755484 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:01.755446 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:04:18.611700 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.611659 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg"] Nov 03 11:04:18.618035 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.618004 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.638407 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.637328 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg"] Nov 03 11:04:18.709472 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.709438 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d9f2ce-openshift-default-dockercfg-zldhk\"" Nov 03 11:04:18.755625 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.755581 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.755812 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.755702 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa80780-52f3-4910-b141-4fc80639cc6e-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.755812 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.755757 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7czv\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-kube-api-access-c7czv\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.755892 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.755868 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.755928 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.755913 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.756020 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.755997 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.756060 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.756041 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.756101 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.756070 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.756101 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.756085 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.857383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.857289 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7czv\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-kube-api-access-c7czv\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.857580 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.857424 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.857580 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.857468 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.857580 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.857495 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.857580 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.857520 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.857580 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.857557 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.857881 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.857582 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.857881 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.857833 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.858523 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.857922 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa80780-52f3-4910-b141-4fc80639cc6e-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.859113 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.859072 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.859212 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.859176 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.859451 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.859426 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.859561 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.859531 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.859723 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.859704 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa80780-52f3-4910-b141-4fc80639cc6e-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.860346 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.860283 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.860907 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.860887 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.865526 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.865474 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.868771 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.868750 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7czv\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-kube-api-access-c7czv\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:18.949952 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:18.949914 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:19.300235 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:19.300199 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg"] Nov 03 11:04:19.301232 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:04:19.301202 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa80780_52f3_4910_b141_4fc80639cc6e.slice/crio-8c31a3742b7aaf6d022202ac4122b449ce77432506ac8e6b3a8bfad9f016f650 WatchSource:0}: Error finding container 8c31a3742b7aaf6d022202ac4122b449ce77432506ac8e6b3a8bfad9f016f650: Status 404 returned error can't find the container with id 8c31a3742b7aaf6d022202ac4122b449ce77432506ac8e6b3a8bfad9f016f650 Nov 03 11:04:19.304339 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:19.304265 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:04:19.304463 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:19.304384 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:04:19.304463 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:19.304433 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:04:20.044160 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:20.044122 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" event={"ID":"3aa80780-52f3-4910-b141-4fc80639cc6e","Type":"ContainerStarted","Data":"5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1"} Nov 03 11:04:20.044611 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:20.044186 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" event={"ID":"3aa80780-52f3-4910-b141-4fc80639cc6e","Type":"ContainerStarted","Data":"8c31a3742b7aaf6d022202ac4122b449ce77432506ac8e6b3a8bfad9f016f650"} Nov 03 11:04:20.069464 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:20.069393 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" podStartSLOduration=2.069371347 podStartE2EDuration="2.069371347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:20.066104199 +0000 UTC m=+2138.050975548" watchObservedRunningTime="2026-11-03 11:04:20.069371347 +0000 UTC m=+2138.054242654" Nov 03 11:04:20.959318 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:20.953685 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:20.959318 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:20.958848 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 11:04:20.959318 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:20.958908 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 11:04:21.951316 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:21.951270 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 11:04:21.951844 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:21.951375 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 11:04:22.950793 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:22.950754 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 11:04:22.950982 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:22.950813 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 11:04:23.950517 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:23.950476 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 11:04:23.950939 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:23.950539 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 11:04:24.302765 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:04:24.302678 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:04:24.959803 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:24.959315 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 11:04:24.959803 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:24.959382 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 11:04:26.180002 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:26.179971 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:26.180784 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:26.180759 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:26.182904 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:04:26.182880 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:04:54.276979 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:04:54.276931 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:05:03.665044 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.665009 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45"] Nov 03 11:05:03.671665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.671632 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.683488 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.683450 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45"] Nov 03 11:05:03.819510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.819469 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.819510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.819515 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593043cd-9a63-4043-8365-d97b79858319-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.819793 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.819541 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.819793 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.819603 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.819793 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.819713 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.819793 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.819749 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.820023 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.819833 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.820023 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.819952 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593043cd-9a63-4043-8365-d97b79858319-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.820151 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.820040 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh6xn\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-kube-api-access-vh6xn\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921246 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921147 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921246 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921212 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593043cd-9a63-4043-8365-d97b79858319-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921246 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921245 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921581 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921269 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921581 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921388 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921581 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921442 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921581 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921502 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921801 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921623 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593043cd-9a63-4043-8365-d97b79858319-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921801 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921673 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.921801 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921760 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh6xn\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-kube-api-access-vh6xn\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.922014 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921986 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.922088 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.921993 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.922160 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.922086 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.922327 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.922274 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593043cd-9a63-4043-8365-d97b79858319-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.924076 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.924046 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.924416 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.924391 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593043cd-9a63-4043-8365-d97b79858319-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.929333 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.929270 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:03.929449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:03.929386 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh6xn\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-kube-api-access-vh6xn\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:04.006184 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:04.006142 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:04.330439 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:04.330405 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45"] Nov 03 11:05:04.345664 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:05:04.334394 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593043cd_9a63_4043_8365_d97b79858319.slice/crio-b0d5e7bff6d9871e2f28843a4fd9fc3014cf418c29d02037438fec76aa756c34 WatchSource:0}: Error finding container b0d5e7bff6d9871e2f28843a4fd9fc3014cf418c29d02037438fec76aa756c34: Status 404 returned error can't find the container with id b0d5e7bff6d9871e2f28843a4fd9fc3014cf418c29d02037438fec76aa756c34 Nov 03 11:05:04.345664 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:04.337381 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:05:04.345664 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:04.337474 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:05:04.345664 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:04.337518 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:05:04.720179 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:04.720035 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" event={"ID":"593043cd-9a63-4043-8365-d97b79858319","Type":"ContainerStarted","Data":"7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed"} Nov 03 11:05:04.720179 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:04.720132 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" event={"ID":"593043cd-9a63-4043-8365-d97b79858319","Type":"ContainerStarted","Data":"b0d5e7bff6d9871e2f28843a4fd9fc3014cf418c29d02037438fec76aa756c34"} Nov 03 11:05:04.739705 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:04.739401 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" podStartSLOduration=1.739383545 podStartE2EDuration="1.739383545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:04.738836011 +0000 UTC m=+2182.723707294" watchObservedRunningTime="2026-11-03 11:05:04.739383545 +0000 UTC m=+2182.724254829" Nov 03 11:05:05.006542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:05.006450 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:05.008222 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:05.008193 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 11:05:05.008400 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:05.008251 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 11:05:06.008274 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:06.008235 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 11:05:06.008744 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:06.008312 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 11:05:07.007282 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:07.007240 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 11:05:07.007498 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:07.007320 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 11:05:08.006545 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:08.006502 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 11:05:08.007038 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:08.006587 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 11:05:09.007231 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:09.007193 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 11:05:09.007668 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:09.007269 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 11:05:10.007276 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:10.007244 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 11:05:10.007712 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:10.007330 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 11:05:11.011685 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:11.011656 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:11.012129 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:11.012112 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:11.012902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:11.012882 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:05:24.419787 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:05:24.419688 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:05:37.461803 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.461767 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn"] Nov 03 11:05:37.467123 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.466913 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.493915 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.493882 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-35936b-serving-cert\"" Nov 03 11:05:37.498976 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.498940 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn"] Nov 03 11:05:37.608278 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.608222 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqkjq\" (UniqueName: \"kubernetes.io/projected/5f032621-ff28-4049-abe9-698a496a82f8-kube-api-access-zqkjq\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.608536 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.608373 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.608536 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.608406 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-metrics-certs\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.608660 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.608579 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f032621-ff28-4049-abe9-698a496a82f8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.710699 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.710663 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqkjq\" (UniqueName: \"kubernetes.io/projected/5f032621-ff28-4049-abe9-698a496a82f8-kube-api-access-zqkjq\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.710904 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.710750 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.710904 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.710774 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-metrics-certs\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.710996 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.710900 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f032621-ff28-4049-abe9-698a496a82f8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.711583 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.711556 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f032621-ff28-4049-abe9-698a496a82f8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.713555 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.713504 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.713555 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.713504 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-metrics-certs\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.719061 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.719030 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqkjq\" (UniqueName: \"kubernetes.io/projected/5f032621-ff28-4049-abe9-698a496a82f8-kube-api-access-zqkjq\") pod \"maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:37.814071 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:37.814022 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:38.104171 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:38.104003 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn"] Nov 03 11:05:38.109845 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:05:38.109812 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f032621_ff28_4049_abe9_698a496a82f8.slice/crio-145e497c3652d0c8a308619fe7b5d15ece4acba43e2b9ab8aac0b48bf55d157b WatchSource:0}: Error finding container 145e497c3652d0c8a308619fe7b5d15ece4acba43e2b9ab8aac0b48bf55d157b: Status 404 returned error can't find the container with id 145e497c3652d0c8a308619fe7b5d15ece4acba43e2b9ab8aac0b48bf55d157b Nov 03 11:05:39.090137 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:39.090100 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" event={"ID":"5f032621-ff28-4049-abe9-698a496a82f8","Type":"ContainerStarted","Data":"a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12"} Nov 03 11:05:39.090584 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:39.090162 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:39.090584 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:39.090180 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" event={"ID":"5f032621-ff28-4049-abe9-698a496a82f8","Type":"ContainerStarted","Data":"145e497c3652d0c8a308619fe7b5d15ece4acba43e2b9ab8aac0b48bf55d157b"} Nov 03 11:05:39.109491 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:39.109433 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" podStartSLOduration=1.627745969 podStartE2EDuration="2.109418102s" totalImagesPullingTime="481.672133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:39.106451851 +0000 UTC m=+2217.091323133" watchObservedRunningTime="2026-11-03 11:05:39.109418102 +0000 UTC m=+2217.094289754" Nov 03 11:05:45.103724 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:05:45.103694 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:05:54.283783 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:05:54.283737 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:06:24.275717 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:06:24.275672 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:06:29.590341 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:29.588354 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9"] Nov 03 11:06:29.590341 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:29.588653 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" containerID="cri-o://5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9" gracePeriod=30 Nov 03 11:06:34.451517 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.451483 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v"] Nov 03 11:06:34.456996 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.456965 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.468268 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.468239 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-5711cf-serving-cert\"" Nov 03 11:06:34.471380 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.469183 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-metrics-certs\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.471668 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.471643 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0602f904-1cbe-4088-9832-0961e0088df9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.471811 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.471796 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86pn\" (UniqueName: \"kubernetes.io/projected/0602f904-1cbe-4088-9832-0961e0088df9-kube-api-access-q86pn\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.472171 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.472145 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.476236 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.476215 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v"] Nov 03 11:06:34.578321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.577617 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-metrics-certs\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.578321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.577758 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0602f904-1cbe-4088-9832-0961e0088df9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.578321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.577806 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86pn\" (UniqueName: \"kubernetes.io/projected/0602f904-1cbe-4088-9832-0961e0088df9-kube-api-access-q86pn\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.578321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.577995 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.578321 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:06:34.578146 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-serving-cert: secret "maas-api-e2e-extmod-b-5711cf-serving-cert" not found Nov 03 11:06:34.578321 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:06:34.578202 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls podName:0602f904-1cbe-4088-9832-0961e0088df9 nodeName:}" failed. No retries permitted until 2026-11-03 11:06:35.078185813 +0000 UTC m=+2273.063057087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls") pod "maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" (UID: "0602f904-1cbe-4088-9832-0961e0088df9") : secret "maas-api-e2e-extmod-b-5711cf-serving-cert" not found Nov 03 11:06:34.584657 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.584624 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0602f904-1cbe-4088-9832-0961e0088df9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.584915 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.584687 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-metrics-certs\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.597347 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.597313 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86pn\" (UniqueName: \"kubernetes.io/projected/0602f904-1cbe-4088-9832-0961e0088df9-kube-api-access-q86pn\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:34.890473 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.890444 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 11:06:34.984780 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.984684 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02267306-b3b0-43d5-a639-e04d7dd016b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02267306-b3b0-43d5-a639-e04d7dd016b6-istiod-ca-cert\") pod \"02267306-b3b0-43d5-a639-e04d7dd016b6\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " Nov 03 11:06:34.984780 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.984741 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-socket\") pod \"02267306-b3b0-43d5-a639-e04d7dd016b6\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " Nov 03 11:06:34.985031 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.984792 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-envoy\") pod \"02267306-b3b0-43d5-a639-e04d7dd016b6\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " Nov 03 11:06:34.985031 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.984867 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-token\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-token\") pod \"02267306-b3b0-43d5-a639-e04d7dd016b6\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " Nov 03 11:06:34.985031 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.984886 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-certs\") pod \"02267306-b3b0-43d5-a639-e04d7dd016b6\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " Nov 03 11:06:34.985185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.985046 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-kube-api-access-g45lm\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-kube-api-access-g45lm\") pod \"02267306-b3b0-43d5-a639-e04d7dd016b6\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " Nov 03 11:06:34.985185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.985142 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-data\") pod \"02267306-b3b0-43d5-a639-e04d7dd016b6\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " Nov 03 11:06:34.985185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.985149 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02267306-b3b0-43d5-a639-e04d7dd016b6-istiod-ca-cert" pod "02267306-b3b0-43d5-a639-e04d7dd016b6" (UID: "02267306-b3b0-43d5-a639-e04d7dd016b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:06:34.985370 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.985201 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-credential-socket\") pod \"02267306-b3b0-43d5-a639-e04d7dd016b6\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " Nov 03 11:06:34.985370 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.985269 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-podinfo\") pod \"02267306-b3b0-43d5-a639-e04d7dd016b6\" (UID: \"02267306-b3b0-43d5-a639-e04d7dd016b6\") " Nov 03 11:06:34.985484 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.985359 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-certs" pod "02267306-b3b0-43d5-a639-e04d7dd016b6" (UID: "02267306-b3b0-43d5-a639-e04d7dd016b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:06:34.985537 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.985504 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-data" pod "02267306-b3b0-43d5-a639-e04d7dd016b6" (UID: "02267306-b3b0-43d5-a639-e04d7dd016b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:06:34.985622 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.985593 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-credential-socket" pod "02267306-b3b0-43d5-a639-e04d7dd016b6" (UID: "02267306-b3b0-43d5-a639-e04d7dd016b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:06:34.985716 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.985687 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-socket" pod "02267306-b3b0-43d5-a639-e04d7dd016b6" (UID: "02267306-b3b0-43d5-a639-e04d7dd016b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:06:34.986801 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.986772 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:06:34.986936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.986804 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:06:34.986936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.986823 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02267306-b3b0-43d5-a639-e04d7dd016b6-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:06:34.986936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.986839 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:06:34.986936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.986854 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:06:34.987915 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.987888 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-envoy" pod "02267306-b3b0-43d5-a639-e04d7dd016b6" (UID: "02267306-b3b0-43d5-a639-e04d7dd016b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:06:34.988497 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.988472 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-kube-api-access-g45lm" pod "02267306-b3b0-43d5-a639-e04d7dd016b6" (UID: "02267306-b3b0-43d5-a639-e04d7dd016b6"). InnerVolumeSpecName "kube-api-access-g45lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:06:34.988603 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.988479 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-podinfo" pod "02267306-b3b0-43d5-a639-e04d7dd016b6" (UID: "02267306-b3b0-43d5-a639-e04d7dd016b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:06:34.988688 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:34.988665 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-token" pod "02267306-b3b0-43d5-a639-e04d7dd016b6" (UID: "02267306-b3b0-43d5-a639-e04d7dd016b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:06:35.088344 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.088263 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:35.088753 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.088731 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:06:35.088829 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.088759 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:06:35.088829 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.088774 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:06:35.088829 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.088791 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45lm\" (UniqueName: \"kubernetes.io/projected/02267306-b3b0-43d5-a639-e04d7dd016b6-kube-api-access-g45lm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:06:35.091924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.091887 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:35.390160 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.390130 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:35.682383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.682179 5086 generic.go:356] "Generic (PLEG): container finished" podID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerID="5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9" exitCode=0 Nov 03 11:06:35.682383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.682223 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" event={"ID":"02267306-b3b0-43d5-a639-e04d7dd016b6","Type":"ContainerDied","Data":"5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9"} Nov 03 11:06:35.682383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.682318 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" Nov 03 11:06:35.683140 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.683105 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9" event={"ID":"02267306-b3b0-43d5-a639-e04d7dd016b6","Type":"ContainerDied","Data":"8d9ab4b6322a1d55646d34c6bbb18dfa0763536cc4a9fe5c65792a1eb808d96e"} Nov 03 11:06:35.683279 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.683158 5086 scope.go:122] "RemoveContainer" containerID="5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9" Nov 03 11:06:35.699147 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.699125 5086 scope.go:122] "RemoveContainer" containerID="5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9" Nov 03 11:06:35.699618 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:06:35.699593 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9\": container with ID starting with 5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9 not found: ID does not exist" containerID="5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9" Nov 03 11:06:35.699710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.699632 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9"} err="failed to get container status \"5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9\": rpc error: code = NotFound desc = could not find container \"5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9\": container with ID starting with 5d09543a3452e19890f4ed9687c6d431deb948b37284cd6a1b4244f6c3ca76c9 not found: ID does not exist" Nov 03 11:06:35.712496 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.712466 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9"] Nov 03 11:06:35.719271 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.719254 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v"] Nov 03 11:06:35.720310 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.720274 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 11:06:35.723682 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:35.723661 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-mj2h9"] Nov 03 11:06:36.176421 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:36.176387 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" path="/var/lib/kubelet/pods/02267306-b3b0-43d5-a639-e04d7dd016b6/volumes" Nov 03 11:06:36.693136 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:36.693092 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" event={"ID":"0602f904-1cbe-4088-9832-0961e0088df9","Type":"ContainerStarted","Data":"c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a"} Nov 03 11:06:36.693597 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:36.693156 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" event={"ID":"0602f904-1cbe-4088-9832-0961e0088df9","Type":"ContainerStarted","Data":"67d8c392a909498d33ad18769e0b4f7e15ec2849a0f425b51825c4ca2d2060f6"} Nov 03 11:06:36.693597 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:36.693180 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:36.716154 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:36.716083 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" podStartSLOduration=2.213746261 podStartE2EDuration="2.716005819s" totalImagesPullingTime="502.259558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:36.713179653 +0000 UTC m=+2274.698050948" watchObservedRunningTime="2026-11-03 11:06:36.716005819 +0000 UTC m=+2274.700877101" Nov 03 11:06:42.707849 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:06:42.707817 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:06:54.305001 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:06:54.304733 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:07:24.293020 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:07:24.292974 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:07:34.280740 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:34.280705 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp"] Nov 03 11:07:34.281143 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:34.280907 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" containerID="cri-o://754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7" gracePeriod=30 Nov 03 11:07:39.659802 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.659769 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 11:07:39.757646 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.757612 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ea01029-a736-4113-b371-eb957f0b566f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea01029-a736-4113-b371-eb957f0b566f-istio-podinfo\") pod \"2ea01029-a736-4113-b371-eb957f0b566f\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " Nov 03 11:07:39.757847 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.757660 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-credential-socket\") pod \"2ea01029-a736-4113-b371-eb957f0b566f\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " Nov 03 11:07:39.757847 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.757773 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-envoy\") pod \"2ea01029-a736-4113-b371-eb957f0b566f\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " Nov 03 11:07:39.757985 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.757874 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-socket\") pod \"2ea01029-a736-4113-b371-eb957f0b566f\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " Nov 03 11:07:39.757985 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.757928 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-certs\") pod \"2ea01029-a736-4113-b371-eb957f0b566f\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " Nov 03 11:07:39.757985 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.757966 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-data\") pod \"2ea01029-a736-4113-b371-eb957f0b566f\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " Nov 03 11:07:39.758156 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.758019 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-istio-token\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-istio-token\") pod \"2ea01029-a736-4113-b371-eb957f0b566f\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " Nov 03 11:07:39.758156 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.758048 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-credential-socket" pod "2ea01029-a736-4113-b371-eb957f0b566f" (UID: "2ea01029-a736-4113-b371-eb957f0b566f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:07:39.758156 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.758093 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea01029-a736-4113-b371-eb957f0b566f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea01029-a736-4113-b371-eb957f0b566f-istiod-ca-cert\") pod \"2ea01029-a736-4113-b371-eb957f0b566f\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " Nov 03 11:07:39.758334 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.758152 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-kube-api-access-jsxn4\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-kube-api-access-jsxn4\") pod \"2ea01029-a736-4113-b371-eb957f0b566f\" (UID: \"2ea01029-a736-4113-b371-eb957f0b566f\") " Nov 03 11:07:39.758334 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.758270 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-socket" pod "2ea01029-a736-4113-b371-eb957f0b566f" (UID: "2ea01029-a736-4113-b371-eb957f0b566f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:07:39.758488 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.758464 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-data" pod "2ea01029-a736-4113-b371-eb957f0b566f" (UID: "2ea01029-a736-4113-b371-eb957f0b566f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:07:39.758928 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.758650 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-certs" pod "2ea01029-a736-4113-b371-eb957f0b566f" (UID: "2ea01029-a736-4113-b371-eb957f0b566f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:07:39.758928 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.758782 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea01029-a736-4113-b371-eb957f0b566f-istiod-ca-cert" pod "2ea01029-a736-4113-b371-eb957f0b566f" (UID: "2ea01029-a736-4113-b371-eb957f0b566f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:07:39.759282 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.759251 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:07:39.759282 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.759277 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:07:39.759477 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.759330 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:07:39.759477 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.759353 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea01029-a736-4113-b371-eb957f0b566f-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:07:39.759477 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.759367 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:07:39.761678 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.761657 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-kube-api-access-jsxn4" pod "2ea01029-a736-4113-b371-eb957f0b566f" (UID: "2ea01029-a736-4113-b371-eb957f0b566f"). InnerVolumeSpecName "kube-api-access-jsxn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:07:39.761809 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.761689 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ea01029-a736-4113-b371-eb957f0b566f-istio-podinfo" pod "2ea01029-a736-4113-b371-eb957f0b566f" (UID: "2ea01029-a736-4113-b371-eb957f0b566f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:07:39.762100 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.762083 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-istio-token" pod "2ea01029-a736-4113-b371-eb957f0b566f" (UID: "2ea01029-a736-4113-b371-eb957f0b566f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:07:39.762240 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.762218 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-envoy" pod "2ea01029-a736-4113-b371-eb957f0b566f" (UID: "2ea01029-a736-4113-b371-eb957f0b566f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:07:39.860496 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.860343 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:07:39.860496 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.860385 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsxn4\" (UniqueName: \"kubernetes.io/projected/2ea01029-a736-4113-b371-eb957f0b566f-kube-api-access-jsxn4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:07:39.860496 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.860400 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea01029-a736-4113-b371-eb957f0b566f-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:07:39.862854 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:39.860414 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea01029-a736-4113-b371-eb957f0b566f-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:07:40.348368 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:40.348330 5086 generic.go:356] "Generic (PLEG): container finished" podID="2ea01029-a736-4113-b371-eb957f0b566f" containerID="754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7" exitCode=0 Nov 03 11:07:40.348589 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:40.348414 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" Nov 03 11:07:40.348589 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:40.348413 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" event={"ID":"2ea01029-a736-4113-b371-eb957f0b566f","Type":"ContainerDied","Data":"754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7"} Nov 03 11:07:40.348589 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:40.348518 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp" event={"ID":"2ea01029-a736-4113-b371-eb957f0b566f","Type":"ContainerDied","Data":"fc80ad30012dabacf914c4cc6f246dc9e3c15b26417a6b849c9bede2347d3935"} Nov 03 11:07:40.348589 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:40.348548 5086 scope.go:122] "RemoveContainer" containerID="754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7" Nov 03 11:07:40.361599 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:40.361578 5086 scope.go:122] "RemoveContainer" containerID="754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7" Nov 03 11:07:40.361954 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:07:40.361908 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7\": container with ID starting with 754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7 not found: ID does not exist" containerID="754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7" Nov 03 11:07:40.362079 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:40.361952 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7"} err="failed to get container status \"754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7\": rpc error: code = NotFound desc = could not find container \"754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7\": container with ID starting with 754d050559c2b9ac5f755fda58b1f77b8b5f420916be8a6e4fd2ced3560f3bf7 not found: ID does not exist" Nov 03 11:07:40.369398 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:40.369372 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp"] Nov 03 11:07:40.374015 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:40.373988 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-jrqjp"] Nov 03 11:07:42.173651 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:07:42.173619 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea01029-a736-4113-b371-eb957f0b566f" path="/var/lib/kubelet/pods/2ea01029-a736-4113-b371-eb957f0b566f/volumes" Nov 03 11:07:54.277019 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:07:54.276973 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:08:24.276095 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:08:24.275996 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:08:43.035019 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:08:43.034989 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:08:43.040201 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:08:43.040175 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:08:43.058407 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:08:43.058381 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:08:43.063644 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:08:43.063619 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:08:54.276257 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:08:54.276213 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:09:24.331074 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:09:24.331023 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:09:54.275784 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:09:54.275695 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:10:12.105330 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.105257 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj"] Nov 03 11:10:12.106070 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.106054 5086 cpu_manager.go:414] "containerMap: removing container" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" Nov 03 11:10:12.106132 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.106071 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" Nov 03 11:10:12.106132 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.106087 5086 cpu_manager.go:414] "containerMap: removing container" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" Nov 03 11:10:12.106132 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.106092 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" Nov 03 11:10:12.106242 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.106216 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea01029-a736-4113-b371-eb957f0b566f" containerName="istio-proxy" Nov 03 11:10:12.106242 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.106228 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="02267306-b3b0-43d5-a639-e04d7dd016b6" containerName="istio-proxy" Nov 03 11:10:12.110770 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.110735 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.124428 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.124393 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj"] Nov 03 11:10:12.199433 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.199395 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.199630 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.199456 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.199683 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.199619 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227wf\" (UniqueName: \"kubernetes.io/projected/bb8055e0-8ed8-4f82-9816-2afed09b243e-kube-api-access-227wf\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.199721 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.199677 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.199864 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.199843 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8055e0-8ed8-4f82-9816-2afed09b243e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.199902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.199879 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.199939 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.199910 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.200131 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.200104 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.200255 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.200240 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.301906 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.301869 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.301920 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.301943 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227wf\" (UniqueName: \"kubernetes.io/projected/bb8055e0-8ed8-4f82-9816-2afed09b243e-kube-api-access-227wf\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.301973 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.302007 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8055e0-8ed8-4f82-9816-2afed09b243e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.302027 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.302062 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302493 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.302214 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302493 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.302306 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302493 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.302406 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302793 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.302456 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.302793 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.302485 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.303010 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.302806 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.303129 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.303106 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8055e0-8ed8-4f82-9816-2afed09b243e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.304766 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.304746 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.304998 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.304979 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.311026 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.310997 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8055e0-8ed8-4f82-9816-2afed09b243e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.311362 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.311342 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227wf\" (UniqueName: \"kubernetes.io/projected/bb8055e0-8ed8-4f82-9816-2afed09b243e-kube-api-access-227wf\") pod \"maas-default-gateway-openshift-default-6cc458d887-n7vfj\" (UID: \"bb8055e0-8ed8-4f82-9816-2afed09b243e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.434793 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.434695 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:12.771648 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.768776 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj"] Nov 03 11:10:12.796701 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.796248 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:10:12.796701 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.796372 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:10:12.796701 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.796424 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:10:12.858259 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:12.858152 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" event={"ID":"bb8055e0-8ed8-4f82-9816-2afed09b243e","Type":"ContainerStarted","Data":"6ac8b056796102d60d2b8c2f7c1ab56da025776826dcafe675f586c34930b9a3"} Nov 03 11:10:13.868524 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:13.868486 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" event={"ID":"bb8055e0-8ed8-4f82-9816-2afed09b243e","Type":"ContainerStarted","Data":"a842dd9495a3bd97abde53d900ce60ed9d0be7e6a72342cddd2dc5fc678830b0"} Nov 03 11:10:13.893236 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:13.893177 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" podStartSLOduration=1.893161468 podStartE2EDuration="1.893161468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:13.89065525 +0000 UTC m=+2491.875526536" watchObservedRunningTime="2026-11-03 11:10:13.893161468 +0000 UTC m=+2491.878032751" Nov 03 11:10:14.451126 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:14.450193 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:14.451126 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:14.451112 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n7vfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 11:10:14.451410 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:14.451168 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" podUID="bb8055e0-8ed8-4f82-9816-2afed09b243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 11:10:15.437548 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:15.437513 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n7vfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 11:10:15.438162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:15.438131 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" podUID="bb8055e0-8ed8-4f82-9816-2afed09b243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 11:10:16.442131 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:16.441660 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n7vfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 11:10:16.442131 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:16.441724 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" podUID="bb8055e0-8ed8-4f82-9816-2afed09b243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 11:10:17.436208 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:17.435732 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n7vfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 11:10:17.436208 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:17.435797 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" podUID="bb8055e0-8ed8-4f82-9816-2afed09b243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 11:10:18.448816 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:18.448772 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n7vfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 11:10:18.449516 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:18.449481 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" podUID="bb8055e0-8ed8-4f82-9816-2afed09b243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 11:10:19.463831 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:19.463787 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n7vfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 11:10:19.526357 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:19.526247 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" podUID="bb8055e0-8ed8-4f82-9816-2afed09b243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 11:10:20.437729 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:20.437685 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n7vfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 11:10:20.437957 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:20.437758 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" podUID="bb8055e0-8ed8-4f82-9816-2afed09b243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 11:10:22.435781 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:22.435737 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n7vfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 11:10:22.436494 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:22.436452 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" podUID="bb8055e0-8ed8-4f82-9816-2afed09b243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 11:10:22.521768 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:22.521720 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:22.524128 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:22.524089 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:22.525605 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:10:22.525584 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n7vfj" Nov 03 11:10:24.288642 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:10:24.288569 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:10:54.278247 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:10:54.278189 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:11:06.755552 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.755520 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w"] Nov 03 11:11:06.760525 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.760507 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:06.773244 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.773215 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w"] Nov 03 11:11:06.914973 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.914933 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:06.915155 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.915083 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:06.915243 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.915226 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:06.915289 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.915277 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978s6\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-kube-api-access-978s6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:06.915289 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.915318 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:06.915442 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.915375 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:06.915519 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:06.915498 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.017313 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.017220 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.017495 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.017354 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.017495 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.017472 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.017598 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.017498 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978s6\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-kube-api-access-978s6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.017598 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.017517 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.017713 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.017670 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.017774 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.017756 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.018177 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.018146 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.020019 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.019986 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.020164 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.020089 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.020164 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.020139 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.020258 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.020193 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.025051 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.025024 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.025392 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.025372 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-978s6\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-kube-api-access-978s6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sb26w\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.077323 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.077251 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.368400 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.368369 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w"] Nov 03 11:11:07.368911 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:11:07.368887 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7979985_a6e6_4e9f_91f9_82d9c7940ed4.slice/crio-53f26061a4e7a0539cd181033ec13354e4f9e16a40d188a377bb103cbfb41b2b WatchSource:0}: Error finding container 53f26061a4e7a0539cd181033ec13354e4f9e16a40d188a377bb103cbfb41b2b: Status 404 returned error can't find the container with id 53f26061a4e7a0539cd181033ec13354e4f9e16a40d188a377bb103cbfb41b2b Nov 03 11:11:07.372391 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.372349 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:11:07.372461 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.372434 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:11:07.717383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.714502 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" event={"ID":"a7979985-a6e6-4e9f-91f9-82d9c7940ed4","Type":"ContainerStarted","Data":"dd12f08e1561d6fd376f5f5fbd25f17234c7b9fb7c991c6f23da75f404ebe60a"} Nov 03 11:11:07.717383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.714574 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" event={"ID":"a7979985-a6e6-4e9f-91f9-82d9c7940ed4","Type":"ContainerStarted","Data":"53f26061a4e7a0539cd181033ec13354e4f9e16a40d188a377bb103cbfb41b2b"} Nov 03 11:11:07.717383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.715808 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:07.748848 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:07.748788 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" podStartSLOduration=1.748771145 podStartE2EDuration="1.748771145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:11:07.74510737 +0000 UTC m=+2545.729978654" watchObservedRunningTime="2026-11-03 11:11:07.748771145 +0000 UTC m=+2545.733642429" Nov 03 11:11:08.726144 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:11:08.726112 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:11:24.275738 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:11:24.275655 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:11:54.278224 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:11:54.278181 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:12:24.275594 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:12:24.275553 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:12:54.275694 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:12:54.275605 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:13:24.283100 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:13:24.283053 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:13:43.083743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:13:43.083714 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:13:43.089043 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:13:43.089018 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:13:43.108585 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:13:43.108559 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:13:43.113527 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:13:43.113502 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:13:54.280172 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:13:54.280127 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:14:24.275795 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:14:24.275704 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:14:54.275522 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:14:54.275472 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:15:14.175156 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.175121 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn"] Nov 03 11:15:14.181506 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.181471 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.196400 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.196358 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-055d82-serving-cert\"" Nov 03 11:15:14.203366 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.203332 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn"] Nov 03 11:15:14.345434 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.343201 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-maas-api-tls\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.345434 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.343277 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-openshift-service-ca\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.345434 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.343599 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-metrics-certs\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.345434 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.343821 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ldfz\" (UniqueName: \"kubernetes.io/projected/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-kube-api-access-9ldfz\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.445468 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.445384 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-metrics-certs\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.445640 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.445513 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldfz\" (UniqueName: \"kubernetes.io/projected/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-kube-api-access-9ldfz\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.445640 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.445610 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-maas-api-tls\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.445727 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.445651 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-openshift-service-ca\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.446414 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.446380 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-openshift-service-ca\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.448190 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.448166 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-maas-api-tls\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.448190 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.448184 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-metrics-certs\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.452837 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.452811 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldfz\" (UniqueName: \"kubernetes.io/projected/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-kube-api-access-9ldfz\") pod \"maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.517047 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.517003 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:14.851319 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.850476 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn"] Nov 03 11:15:14.883909 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:14.883883 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 11:15:15.648074 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:15.647979 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" event={"ID":"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738","Type":"ContainerStarted","Data":"a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4"} Nov 03 11:15:15.648074 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:15.648035 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" event={"ID":"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738","Type":"ContainerStarted","Data":"fb251fa34f07ef16d5b0e7e5b5acb242587c65ac238718fcf1de4d099dca800f"} Nov 03 11:15:15.648074 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:15.648056 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:15.668450 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:15.668391 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" podStartSLOduration=1.227866901 podStartE2EDuration="1.668376232s" totalImagesPullingTime="440.509331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:15.664729805 +0000 UTC m=+2793.649601088" watchObservedRunningTime="2026-11-03 11:15:15.668376232 +0000 UTC m=+2793.653247516" Nov 03 11:15:21.662636 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:21.662605 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:15:24.276585 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:15:24.276544 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:15:38.777120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:38.777073 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh"] Nov 03 11:15:38.806316 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:38.799856 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:38.809369 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:38.806734 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3eb43f-serving-cert\"" Nov 03 11:15:38.819329 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:38.816871 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh"] Nov 03 11:15:38.922537 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:38.922490 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:38.922814 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:38.922798 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-metrics-certs\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:38.923015 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:38.923000 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb691805-f85b-4e37-b201-0dd3137389b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:38.923388 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:38.923365 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7thg\" (UniqueName: \"kubernetes.io/projected/eb691805-f85b-4e37-b201-0dd3137389b6-kube-api-access-l7thg\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.024498 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.024456 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7thg\" (UniqueName: \"kubernetes.io/projected/eb691805-f85b-4e37-b201-0dd3137389b6-kube-api-access-l7thg\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.024732 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.024575 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.024732 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.024603 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-metrics-certs\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.024732 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.024678 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb691805-f85b-4e37-b201-0dd3137389b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.024732 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:15:39.024723 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-serving-cert: secret "maas-api-e2e-inf-b-3eb43f-serving-cert" not found Nov 03 11:15:39.024949 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:15:39.024790 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls podName:eb691805-f85b-4e37-b201-0dd3137389b6 nodeName:}" failed. No retries permitted until 2026-11-03 11:15:39.524768588 +0000 UTC m=+2817.509639871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls") pod "maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" (UID: "eb691805-f85b-4e37-b201-0dd3137389b6") : secret "maas-api-e2e-inf-b-3eb43f-serving-cert" not found Nov 03 11:15:39.025504 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.025479 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb691805-f85b-4e37-b201-0dd3137389b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.027670 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.027603 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-metrics-certs\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.034252 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.034213 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7thg\" (UniqueName: \"kubernetes.io/projected/eb691805-f85b-4e37-b201-0dd3137389b6-kube-api-access-l7thg\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.534730 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.534687 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.537208 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.537184 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:39.831974 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:39.830721 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:40.179761 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:40.179731 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh"] Nov 03 11:15:40.180968 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:15:40.180944 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb691805_f85b_4e37_b201_0dd3137389b6.slice/crio-bf0de88f334a75d3fd1915efd3f122199290dde1703184ed33fa56796917e0a0 WatchSource:0}: Error finding container bf0de88f334a75d3fd1915efd3f122199290dde1703184ed33fa56796917e0a0: Status 404 returned error can't find the container with id bf0de88f334a75d3fd1915efd3f122199290dde1703184ed33fa56796917e0a0 Nov 03 11:15:40.953090 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:40.952999 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" event={"ID":"eb691805-f85b-4e37-b201-0dd3137389b6","Type":"ContainerStarted","Data":"7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9"} Nov 03 11:15:40.953090 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:40.953066 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" event={"ID":"eb691805-f85b-4e37-b201-0dd3137389b6","Type":"ContainerStarted","Data":"bf0de88f334a75d3fd1915efd3f122199290dde1703184ed33fa56796917e0a0"} Nov 03 11:15:40.953540 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:40.953094 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:40.971004 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:40.970924 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" podStartSLOduration=2.5319554220000002 podStartE2EDuration="2.970906812s" totalImagesPullingTime="438.95139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:40.970336325 +0000 UTC m=+2818.955207609" watchObservedRunningTime="2026-11-03 11:15:40.970906812 +0000 UTC m=+2818.955778096" Nov 03 11:15:46.968392 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:15:46.968364 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:15:54.280554 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:15:54.280445 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:16:02.978724 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:02.978687 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg"] Nov 03 11:16:02.979131 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:02.978897 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" containerID="cri-o://a6273df66ad5053a9eb0b17d8f5da960e62743223aa41e2eed6d9fde671a55e5" gracePeriod=30 Nov 03 11:16:04.974281 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:04.974240 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb"] Nov 03 11:16:04.974753 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:04.974464 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" containerID="cri-o://e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65" gracePeriod=30 Nov 03 11:16:08.363538 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.363494 5086 generic.go:356] "Generic (PLEG): container finished" podID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerID="a6273df66ad5053a9eb0b17d8f5da960e62743223aa41e2eed6d9fde671a55e5" exitCode=0 Nov 03 11:16:08.365230 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.363565 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" event={"ID":"74f5f3e8-9726-44ef-9bff-68e68c8e35b8","Type":"ContainerDied","Data":"a6273df66ad5053a9eb0b17d8f5da960e62743223aa41e2eed6d9fde671a55e5"} Nov 03 11:16:08.398757 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.398727 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 11:16:08.442394 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.442351 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-podinfo\") pod \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " Nov 03 11:16:08.442592 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.442427 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-kube-api-access-sqgmp\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-kube-api-access-sqgmp\") pod \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " Nov 03 11:16:08.442592 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.442472 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-data\") pod \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " Nov 03 11:16:08.442899 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.442855 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-data" pod "74f5f3e8-9726-44ef-9bff-68e68c8e35b8" (UID: "74f5f3e8-9726-44ef-9bff-68e68c8e35b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:08.443166 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.443138 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-certs\") pod \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " Nov 03 11:16:08.443341 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.443180 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-token\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-token\") pod \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " Nov 03 11:16:08.443528 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.443490 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-certs" pod "74f5f3e8-9726-44ef-9bff-68e68c8e35b8" (UID: "74f5f3e8-9726-44ef-9bff-68e68c8e35b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:08.443676 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.443656 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-envoy\") pod \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " Nov 03 11:16:08.443877 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.443858 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istiod-ca-cert\") pod \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " Nov 03 11:16:08.444457 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.444339 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istiod-ca-cert" pod "74f5f3e8-9726-44ef-9bff-68e68c8e35b8" (UID: "74f5f3e8-9726-44ef-9bff-68e68c8e35b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:16:08.444652 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.444634 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-socket\") pod \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " Nov 03 11:16:08.445158 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.445044 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-socket" pod "74f5f3e8-9726-44ef-9bff-68e68c8e35b8" (UID: "74f5f3e8-9726-44ef-9bff-68e68c8e35b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:08.445340 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.445319 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-credential-socket\") pod \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\" (UID: \"74f5f3e8-9726-44ef-9bff-68e68c8e35b8\") " Nov 03 11:16:08.445556 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.445534 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-credential-socket" pod "74f5f3e8-9726-44ef-9bff-68e68c8e35b8" (UID: "74f5f3e8-9726-44ef-9bff-68e68c8e35b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:08.446314 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.446273 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:08.446451 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.446428 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:08.446556 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.446454 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:08.446556 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.446469 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:08.446556 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.446483 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:08.448174 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.447396 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-kube-api-access-sqgmp" pod "74f5f3e8-9726-44ef-9bff-68e68c8e35b8" (UID: "74f5f3e8-9726-44ef-9bff-68e68c8e35b8"). InnerVolumeSpecName "kube-api-access-sqgmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:16:08.448174 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.447698 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-envoy" pod "74f5f3e8-9726-44ef-9bff-68e68c8e35b8" (UID: "74f5f3e8-9726-44ef-9bff-68e68c8e35b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:08.448527 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.448505 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-podinfo" pod "74f5f3e8-9726-44ef-9bff-68e68c8e35b8" (UID: "74f5f3e8-9726-44ef-9bff-68e68c8e35b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:16:08.448623 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.448608 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-token" pod "74f5f3e8-9726-44ef-9bff-68e68c8e35b8" (UID: "74f5f3e8-9726-44ef-9bff-68e68c8e35b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:16:08.548066 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.547990 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:08.548066 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.548020 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqgmp\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-kube-api-access-sqgmp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:08.548066 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.548031 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:08.548066 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:08.548039 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f5f3e8-9726-44ef-9bff-68e68c8e35b8-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:09.371682 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:09.371653 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" Nov 03 11:16:09.371682 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:09.371670 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg" event={"ID":"74f5f3e8-9726-44ef-9bff-68e68c8e35b8","Type":"ContainerDied","Data":"35cf508936b9e27a9f5239f40c0267ec5f4835f5451b54b208f071f34092c5cd"} Nov 03 11:16:09.372214 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:09.371725 5086 scope.go:122] "RemoveContainer" containerID="a6273df66ad5053a9eb0b17d8f5da960e62743223aa41e2eed6d9fde671a55e5" Nov 03 11:16:09.398711 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:09.398679 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg"] Nov 03 11:16:09.402734 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:09.402706 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-srrmg"] Nov 03 11:16:10.178737 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.178699 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" path="/var/lib/kubelet/pods/74f5f3e8-9726-44ef-9bff-68e68c8e35b8/volumes" Nov 03 11:16:10.270728 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.270704 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 11:16:10.381673 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.381636 5086 generic.go:356] "Generic (PLEG): container finished" podID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerID="e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65" exitCode=0 Nov 03 11:16:10.382097 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.381689 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" Nov 03 11:16:10.382097 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.381715 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" event={"ID":"3146c2f2-37ed-4205-b4e1-ccb52a24f433","Type":"ContainerDied","Data":"e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65"} Nov 03 11:16:10.382097 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.381772 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb" event={"ID":"3146c2f2-37ed-4205-b4e1-ccb52a24f433","Type":"ContainerDied","Data":"60bef3ff74d931e1f1fc88face661065733829400d69a497745ae8e320444cda"} Nov 03 11:16:10.382097 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.381798 5086 scope.go:122] "RemoveContainer" containerID="e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65" Nov 03 11:16:10.393139 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.393120 5086 scope.go:122] "RemoveContainer" containerID="e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65" Nov 03 11:16:10.393461 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:16:10.393436 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65\": container with ID starting with e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65 not found: ID does not exist" containerID="e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65" Nov 03 11:16:10.393547 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.393473 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65"} err="failed to get container status \"e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65\": rpc error: code = NotFound desc = could not find container \"e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65\": container with ID starting with e341387c6c6cbd8345ffe94c128f378061f6e4f8df8cfb994e651b1a59ed1a65 not found: ID does not exist" Nov 03 11:16:10.397769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.397033 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-data\") pod \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " Nov 03 11:16:10.397769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.397103 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-token\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-token\") pod \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " Nov 03 11:16:10.397769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.397192 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-kube-api-access-2hjcw\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-kube-api-access-2hjcw\") pod \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " Nov 03 11:16:10.397769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.397227 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-socket\") pod \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " Nov 03 11:16:10.397769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.397266 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istiod-ca-cert\") pod \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " Nov 03 11:16:10.397769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.397367 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-envoy\") pod \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " Nov 03 11:16:10.397769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.397460 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-podinfo\") pod \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " Nov 03 11:16:10.397769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.397555 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-certs\") pod \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " Nov 03 11:16:10.397769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.397686 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-credential-socket\") pod \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\" (UID: \"3146c2f2-37ed-4205-b4e1-ccb52a24f433\") " Nov 03 11:16:10.399283 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.399259 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-socket" pod "3146c2f2-37ed-4205-b4e1-ccb52a24f433" (UID: "3146c2f2-37ed-4205-b4e1-ccb52a24f433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:10.399471 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.399283 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-credential-socket" pod "3146c2f2-37ed-4205-b4e1-ccb52a24f433" (UID: "3146c2f2-37ed-4205-b4e1-ccb52a24f433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:10.404217 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.404116 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-certs" pod "3146c2f2-37ed-4205-b4e1-ccb52a24f433" (UID: "3146c2f2-37ed-4205-b4e1-ccb52a24f433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:10.404217 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.404194 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-data" pod "3146c2f2-37ed-4205-b4e1-ccb52a24f433" (UID: "3146c2f2-37ed-4205-b4e1-ccb52a24f433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:10.404747 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.404725 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istiod-ca-cert" pod "3146c2f2-37ed-4205-b4e1-ccb52a24f433" (UID: "3146c2f2-37ed-4205-b4e1-ccb52a24f433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:16:10.409775 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.409745 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-podinfo" pod "3146c2f2-37ed-4205-b4e1-ccb52a24f433" (UID: "3146c2f2-37ed-4205-b4e1-ccb52a24f433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:16:10.409775 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.409762 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-token" pod "3146c2f2-37ed-4205-b4e1-ccb52a24f433" (UID: "3146c2f2-37ed-4205-b4e1-ccb52a24f433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:16:10.409775 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.409750 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-envoy" pod "3146c2f2-37ed-4205-b4e1-ccb52a24f433" (UID: "3146c2f2-37ed-4205-b4e1-ccb52a24f433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:16:10.409952 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.409870 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-kube-api-access-2hjcw" pod "3146c2f2-37ed-4205-b4e1-ccb52a24f433" (UID: "3146c2f2-37ed-4205-b4e1-ccb52a24f433"). InnerVolumeSpecName "kube-api-access-2hjcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:16:10.500096 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.500006 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hjcw\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-kube-api-access-2hjcw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:10.500096 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.500039 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:10.500096 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.500050 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:10.500096 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.500059 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:10.500096 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.500069 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:10.500096 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.500088 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:10.500096 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.500098 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:10.500463 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.500107 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:10.500463 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.500144 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3146c2f2-37ed-4205-b4e1-ccb52a24f433-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:16:10.707269 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.707232 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb"] Nov 03 11:16:10.715437 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:10.715406 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-bl8fb"] Nov 03 11:16:12.173212 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:12.173174 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" path="/var/lib/kubelet/pods/3146c2f2-37ed-4205-b4e1-ccb52a24f433/volumes" Nov 03 11:16:13.363868 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.363836 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf"] Nov 03 11:16:13.364684 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.364667 5086 cpu_manager.go:414] "containerMap: removing container" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" Nov 03 11:16:13.364749 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.364686 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" Nov 03 11:16:13.364749 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.364718 5086 cpu_manager.go:414] "containerMap: removing container" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" Nov 03 11:16:13.364749 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.364723 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" Nov 03 11:16:13.364858 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.364833 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f5f3e8-9726-44ef-9bff-68e68c8e35b8" containerName="istio-proxy" Nov 03 11:16:13.364858 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.364844 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="3146c2f2-37ed-4205-b4e1-ccb52a24f433" containerName="istio-proxy" Nov 03 11:16:13.370083 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.370060 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.379081 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.379053 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-24376f-serving-cert\"" Nov 03 11:16:13.381183 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.381155 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf"] Nov 03 11:16:13.444466 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.444433 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9cj5\" (UniqueName: \"kubernetes.io/projected/5c395d47-3fca-4469-86b6-8ffd5f53234a-kube-api-access-z9cj5\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.444648 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.444538 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c395d47-3fca-4469-86b6-8ffd5f53234a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.444648 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.444579 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-metrics-certs\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.444737 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.444699 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.556315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.550132 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-metrics-certs\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.556315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.550443 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.556315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.550844 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9cj5\" (UniqueName: \"kubernetes.io/projected/5c395d47-3fca-4469-86b6-8ffd5f53234a-kube-api-access-z9cj5\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.556315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.550934 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c395d47-3fca-4469-86b6-8ffd5f53234a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.556315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.552405 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c395d47-3fca-4469-86b6-8ffd5f53234a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.556315 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:16:13.552565 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-serving-cert: secret "maas-api-e2e-inf-b-24376f-serving-cert" not found Nov 03 11:16:13.556315 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:16:13.552652 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls podName:5c395d47-3fca-4469-86b6-8ffd5f53234a nodeName:}" failed. No retries permitted until 2026-11-03 11:16:14.052630842 +0000 UTC m=+2852.037502118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls") pod "maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" (UID: "5c395d47-3fca-4469-86b6-8ffd5f53234a") : secret "maas-api-e2e-inf-b-24376f-serving-cert" not found Nov 03 11:16:13.556884 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.556863 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-metrics-certs\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:13.575291 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:13.575259 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9cj5\" (UniqueName: \"kubernetes.io/projected/5c395d47-3fca-4469-86b6-8ffd5f53234a-kube-api-access-z9cj5\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:14.061236 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:14.061188 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:14.063839 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:14.063809 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls\") pod \"maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:14.299831 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:14.299795 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:14.586836 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:14.586799 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf"] Nov 03 11:16:14.587255 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:16:14.587056 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c395d47_3fca_4469_86b6_8ffd5f53234a.slice/crio-daf439fc9b5fbba1e62c899c63da4d13e955bacdddb8d26f2b87c2ef15d45033 WatchSource:0}: Error finding container daf439fc9b5fbba1e62c899c63da4d13e955bacdddb8d26f2b87c2ef15d45033: Status 404 returned error can't find the container with id daf439fc9b5fbba1e62c899c63da4d13e955bacdddb8d26f2b87c2ef15d45033 Nov 03 11:16:15.430410 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:15.430362 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" event={"ID":"5c395d47-3fca-4469-86b6-8ffd5f53234a","Type":"ContainerStarted","Data":"104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6"} Nov 03 11:16:15.430611 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:15.430427 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" event={"ID":"5c395d47-3fca-4469-86b6-8ffd5f53234a","Type":"ContainerStarted","Data":"daf439fc9b5fbba1e62c899c63da4d13e955bacdddb8d26f2b87c2ef15d45033"} Nov 03 11:16:15.430611 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:15.430457 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:15.452822 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:15.452767 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" podStartSLOduration=2.034198263 podStartE2EDuration="2.452750339s" totalImagesPullingTime="418.552076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:15.450952084 +0000 UTC m=+2853.435823379" watchObservedRunningTime="2026-11-03 11:16:15.452750339 +0000 UTC m=+2853.437621624" Nov 03 11:16:21.445844 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:21.445816 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:16:24.318527 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:16:24.318486 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:16:38.909901 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:38.909864 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l"] Nov 03 11:16:38.916663 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:38.916636 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:38.921117 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:38.921087 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-68aa46-serving-cert\"" Nov 03 11:16:38.927595 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:38.927565 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l"] Nov 03 11:16:38.964442 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:38.964399 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5htj\" (UniqueName: \"kubernetes.io/projected/33d6f189-277a-47de-9410-2c52d9edea84-kube-api-access-q5htj\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:38.964638 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:38.964470 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:38.964638 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:38.964603 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d6f189-277a-47de-9410-2c52d9edea84-openshift-service-ca\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:38.964750 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:38.964653 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-metrics-certs\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.066886 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.066406 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d6f189-277a-47de-9410-2c52d9edea84-openshift-service-ca\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.066886 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.066472 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-metrics-certs\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.066886 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.066727 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5htj\" (UniqueName: \"kubernetes.io/projected/33d6f189-277a-47de-9410-2c52d9edea84-kube-api-access-q5htj\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.066886 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.066788 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.067355 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:16:39.067032 5086 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-serving-cert: secret "maas-api-e2e-inf-a-68aa46-serving-cert" not found Nov 03 11:16:39.067355 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:16:39.067152 5086 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls podName:33d6f189-277a-47de-9410-2c52d9edea84 nodeName:}" failed. No retries permitted until 2026-11-03 11:16:39.567125581 +0000 UTC m=+2877.551996864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls") pod "maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" (UID: "33d6f189-277a-47de-9410-2c52d9edea84") : secret "maas-api-e2e-inf-a-68aa46-serving-cert" not found Nov 03 11:16:39.068830 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.068714 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d6f189-277a-47de-9410-2c52d9edea84-openshift-service-ca\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.070635 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.070614 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-metrics-certs\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.077693 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.077670 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5htj\" (UniqueName: \"kubernetes.io/projected/33d6f189-277a-47de-9410-2c52d9edea84-kube-api-access-q5htj\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.574976 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.574932 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.577879 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.577849 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls\") pod \"maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:39.859037 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:39.858943 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:40.179497 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:16:40.179465 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d6f189_277a_47de_9410_2c52d9edea84.slice/crio-963a8a48acecb41a3da4105261dfd30ad37978428be7c8ad26919ce42a18530e WatchSource:0}: Error finding container 963a8a48acecb41a3da4105261dfd30ad37978428be7c8ad26919ce42a18530e: Status 404 returned error can't find the container with id 963a8a48acecb41a3da4105261dfd30ad37978428be7c8ad26919ce42a18530e Nov 03 11:16:40.180943 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:40.180910 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l"] Nov 03 11:16:40.635735 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:40.635696 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" event={"ID":"33d6f189-277a-47de-9410-2c52d9edea84","Type":"ContainerStarted","Data":"963a8a48acecb41a3da4105261dfd30ad37978428be7c8ad26919ce42a18530e"} Nov 03 11:16:41.646887 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:41.646843 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" event={"ID":"33d6f189-277a-47de-9410-2c52d9edea84","Type":"ContainerStarted","Data":"e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19"} Nov 03 11:16:41.647449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:41.646917 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:47.662085 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:47.662054 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:16:47.683793 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:16:47.683727 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" podStartSLOduration=9.189031082 podStartE2EDuration="9.683709955s" totalImagesPullingTime="494.678873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:41.664418024 +0000 UTC m=+2879.649289303" watchObservedRunningTime="2026-11-03 11:16:47.683709955 +0000 UTC m=+2885.668581239" Nov 03 11:16:54.278751 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:16:54.278707 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:17:15.794972 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:15.794932 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4"] Nov 03 11:17:15.795575 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:15.795156 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" containerID="cri-o://3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070" gracePeriod=30 Nov 03 11:17:20.009187 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:20.009146 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" podUID="bb77e2dfdbc4a16f3bcede022ac475a7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Nov 03 11:17:21.080803 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.080732 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 11:17:21.128910 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.128869 5086 generic.go:356] "Generic (PLEG): container finished" podID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerID="3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070" exitCode=0 Nov 03 11:17:21.129116 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.128932 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" Nov 03 11:17:21.129116 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.128957 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" event={"ID":"42d1ce75-5d3e-440f-8479-5ab13fa736e5","Type":"ContainerDied","Data":"3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070"} Nov 03 11:17:21.129116 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.129015 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4" event={"ID":"42d1ce75-5d3e-440f-8479-5ab13fa736e5","Type":"ContainerDied","Data":"12a75edeed993c85d3843b05da79ea11c33ba1e89b34dfc373da1485dd351842"} Nov 03 11:17:21.129116 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.129038 5086 scope.go:122] "RemoveContainer" containerID="3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070" Nov 03 11:17:21.140662 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.140640 5086 scope.go:122] "RemoveContainer" containerID="3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070" Nov 03 11:17:21.140951 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:17:21.140930 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070\": container with ID starting with 3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070 not found: ID does not exist" containerID="3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070" Nov 03 11:17:21.141016 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.140960 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070"} err="failed to get container status \"3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070\": rpc error: code = NotFound desc = could not find container \"3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070\": container with ID starting with 3294c92b481475953391fdffee3014290f10f13ff60505dd1382a7ab9807c070 not found: ID does not exist" Nov 03 11:17:21.178564 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178532 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-certs\") pod \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " Nov 03 11:17:21.178705 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178603 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-credential-socket\") pod \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " Nov 03 11:17:21.178705 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178650 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-podinfo\") pod \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " Nov 03 11:17:21.178778 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178704 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-token\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-token\") pod \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " Nov 03 11:17:21.178826 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178773 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-socket\") pod \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " Nov 03 11:17:21.178895 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178812 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-certs" pod "42d1ce75-5d3e-440f-8479-5ab13fa736e5" (UID: "42d1ce75-5d3e-440f-8479-5ab13fa736e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:17:21.178895 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178835 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-envoy\") pod \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " Nov 03 11:17:21.178895 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178887 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-data\") pod \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " Nov 03 11:17:21.179060 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178955 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-credential-socket" pod "42d1ce75-5d3e-440f-8479-5ab13fa736e5" (UID: "42d1ce75-5d3e-440f-8479-5ab13fa736e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:17:21.179060 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.178985 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istiod-ca-cert\") pod \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " Nov 03 11:17:21.179060 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.179026 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-kube-api-access-54vmw\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-kube-api-access-54vmw\") pod \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\" (UID: \"42d1ce75-5d3e-440f-8479-5ab13fa736e5\") " Nov 03 11:17:21.179212 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.179138 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-socket" pod "42d1ce75-5d3e-440f-8479-5ab13fa736e5" (UID: "42d1ce75-5d3e-440f-8479-5ab13fa736e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:17:21.179268 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.179251 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-data" pod "42d1ce75-5d3e-440f-8479-5ab13fa736e5" (UID: "42d1ce75-5d3e-440f-8479-5ab13fa736e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:17:21.179498 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.179467 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istiod-ca-cert" pod "42d1ce75-5d3e-440f-8479-5ab13fa736e5" (UID: "42d1ce75-5d3e-440f-8479-5ab13fa736e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:17:21.179902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.179844 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:17:21.179902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.179869 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:17:21.179902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.179885 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:17:21.179902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.179900 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:17:21.180179 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.179913 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:17:21.181188 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.181163 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-podinfo" pod "42d1ce75-5d3e-440f-8479-5ab13fa736e5" (UID: "42d1ce75-5d3e-440f-8479-5ab13fa736e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:17:21.181287 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.181251 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-envoy" pod "42d1ce75-5d3e-440f-8479-5ab13fa736e5" (UID: "42d1ce75-5d3e-440f-8479-5ab13fa736e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:17:21.181429 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.181410 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-token" pod "42d1ce75-5d3e-440f-8479-5ab13fa736e5" (UID: "42d1ce75-5d3e-440f-8479-5ab13fa736e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:17:21.181499 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.181443 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-kube-api-access-54vmw" pod "42d1ce75-5d3e-440f-8479-5ab13fa736e5" (UID: "42d1ce75-5d3e-440f-8479-5ab13fa736e5"). InnerVolumeSpecName "kube-api-access-54vmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:17:21.281891 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.281856 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:17:21.281891 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.281887 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vmw\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-kube-api-access-54vmw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:17:21.281891 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.281897 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:17:21.282146 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.281907 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d1ce75-5d3e-440f-8479-5ab13fa736e5-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:17:21.455109 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.455065 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4"] Nov 03 11:17:21.458824 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:21.458798 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-fl9r4"] Nov 03 11:17:22.211377 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:22.211340 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" path="/var/lib/kubelet/pods/42d1ce75-5d3e-440f-8479-5ab13fa736e5/volumes" Nov 03 11:17:24.276745 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:17:24.276655 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:17:25.063149 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.063110 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d"] Nov 03 11:17:25.064042 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.064019 5086 cpu_manager.go:414] "containerMap: removing container" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" Nov 03 11:17:25.064164 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.064042 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" Nov 03 11:17:25.064329 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.064281 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d1ce75-5d3e-440f-8479-5ab13fa736e5" containerName="istio-proxy" Nov 03 11:17:25.077137 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.077106 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d"] Nov 03 11:17:25.077386 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.077200 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.085649 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.085615 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b13404-serving-cert\"" Nov 03 11:17:25.150202 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.150155 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.150392 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.150339 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6t7k\" (UniqueName: \"kubernetes.io/projected/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-kube-api-access-w6t7k\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.150392 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.150380 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-metrics-certs\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.150525 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.150508 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-maas-api-tls\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.254434 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.254397 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-metrics-certs\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.263103 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.263066 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-maas-api-tls\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.263623 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.263601 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.263848 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.263832 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6t7k\" (UniqueName: \"kubernetes.io/projected/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-kube-api-access-w6t7k\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.265136 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.265112 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.281357 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.276901 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-maas-api-tls\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.282120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.282093 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-metrics-certs\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.282325 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.282277 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6t7k\" (UniqueName: \"kubernetes.io/projected/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-kube-api-access-w6t7k\") pod \"maas-api-e2e-inf-a-b13404-75884796cd-v4q2d\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.399829 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.399786 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:25.686602 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:17:25.686569 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c1b8dd_1f29_47a3_8a2b_a2f56526131c.slice/crio-79eb27fd2dad89241036d87963cf000c63f44f12606bff998ac7e2e0ac89347e WatchSource:0}: Error finding container 79eb27fd2dad89241036d87963cf000c63f44f12606bff998ac7e2e0ac89347e: Status 404 returned error can't find the container with id 79eb27fd2dad89241036d87963cf000c63f44f12606bff998ac7e2e0ac89347e Nov 03 11:17:25.691131 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:25.691101 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d"] Nov 03 11:17:26.176819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:26.176780 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" event={"ID":"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c","Type":"ContainerStarted","Data":"79eb27fd2dad89241036d87963cf000c63f44f12606bff998ac7e2e0ac89347e"} Nov 03 11:17:27.186570 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:27.186530 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" event={"ID":"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c","Type":"ContainerStarted","Data":"7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2"} Nov 03 11:17:27.186977 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:27.186613 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:27.205965 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:27.205921 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" podStartSLOduration=1.6377062279999999 podStartE2EDuration="2.205903869s" totalImagesPullingTime="568.197641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:17:27.202165377 +0000 UTC m=+2925.187036660" watchObservedRunningTime="2026-11-03 11:17:27.205903869 +0000 UTC m=+2925.190775151" Nov 03 11:17:33.199932 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:17:33.199901 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:17:54.275877 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:17:54.275836 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:18:24.277865 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:18:24.277828 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:18:43.131314 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:18:43.131275 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:18:43.136373 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:18:43.136351 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:18:43.156781 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:18:43.156759 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:18:43.161663 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:18:43.161646 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:18:54.275316 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:18:54.275214 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:19:24.275169 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:19:24.275129 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:19:54.276035 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:19:54.275991 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:20:24.275696 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:20:24.275606 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:20:54.278012 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:20:54.277975 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:21:19.968215 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:19.968177 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45"] Nov 03 11:21:19.970873 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:19.968413 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" containerID="cri-o://7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed" gracePeriod=30 Nov 03 11:21:24.276415 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:21:24.276378 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:21:25.306084 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.306057 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:21:25.376704 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.376667 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-credential-socket\") pod \"593043cd-9a63-4043-8365-d97b79858319\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " Nov 03 11:21:25.376878 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.376822 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-istio-token\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-istio-token\") pod \"593043cd-9a63-4043-8365-d97b79858319\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " Nov 03 11:21:25.376930 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.376915 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-data\") pod \"593043cd-9a63-4043-8365-d97b79858319\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " Nov 03 11:21:25.376974 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.376942 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-credential-socket" pod "593043cd-9a63-4043-8365-d97b79858319" (UID: "593043cd-9a63-4043-8365-d97b79858319"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:21:25.376974 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.376955 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-certs\") pod \"593043cd-9a63-4043-8365-d97b79858319\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " Nov 03 11:21:25.377067 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.377030 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-envoy\") pod \"593043cd-9a63-4043-8365-d97b79858319\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " Nov 03 11:21:25.377117 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.377063 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/593043cd-9a63-4043-8365-d97b79858319-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593043cd-9a63-4043-8365-d97b79858319-istio-podinfo\") pod \"593043cd-9a63-4043-8365-d97b79858319\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " Nov 03 11:21:25.377459 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.377226 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593043cd-9a63-4043-8365-d97b79858319-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593043cd-9a63-4043-8365-d97b79858319-istiod-ca-cert\") pod \"593043cd-9a63-4043-8365-d97b79858319\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " Nov 03 11:21:25.377459 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.377236 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-certs" pod "593043cd-9a63-4043-8365-d97b79858319" (UID: "593043cd-9a63-4043-8365-d97b79858319"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:21:25.377459 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.377247 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-data" pod "593043cd-9a63-4043-8365-d97b79858319" (UID: "593043cd-9a63-4043-8365-d97b79858319"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:21:25.377459 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.377265 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-socket\") pod \"593043cd-9a63-4043-8365-d97b79858319\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " Nov 03 11:21:25.377459 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.377333 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-kube-api-access-vh6xn\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-kube-api-access-vh6xn\") pod \"593043cd-9a63-4043-8365-d97b79858319\" (UID: \"593043cd-9a63-4043-8365-d97b79858319\") " Nov 03 11:21:25.377802 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.377598 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593043cd-9a63-4043-8365-d97b79858319-istiod-ca-cert" pod "593043cd-9a63-4043-8365-d97b79858319" (UID: "593043cd-9a63-4043-8365-d97b79858319"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:21:25.377876 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.377847 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-socket" pod "593043cd-9a63-4043-8365-d97b79858319" (UID: "593043cd-9a63-4043-8365-d97b79858319"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:21:25.378173 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.378085 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:25.378173 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.378109 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:25.378173 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.378123 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593043cd-9a63-4043-8365-d97b79858319-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:25.378173 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.378137 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:25.378173 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.378151 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:25.380066 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.380014 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/593043cd-9a63-4043-8365-d97b79858319-istio-podinfo" pod "593043cd-9a63-4043-8365-d97b79858319" (UID: "593043cd-9a63-4043-8365-d97b79858319"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:21:25.380391 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.380369 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-envoy" pod "593043cd-9a63-4043-8365-d97b79858319" (UID: "593043cd-9a63-4043-8365-d97b79858319"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:21:25.380391 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.380265 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-istio-token" pod "593043cd-9a63-4043-8365-d97b79858319" (UID: "593043cd-9a63-4043-8365-d97b79858319"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:21:25.383933 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.383892 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-kube-api-access-vh6xn" pod "593043cd-9a63-4043-8365-d97b79858319" (UID: "593043cd-9a63-4043-8365-d97b79858319"). InnerVolumeSpecName "kube-api-access-vh6xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:21:25.479076 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.479042 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh6xn\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-kube-api-access-vh6xn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:25.479076 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.479073 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593043cd-9a63-4043-8365-d97b79858319-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:25.479076 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.479083 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593043cd-9a63-4043-8365-d97b79858319-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:25.479323 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.479092 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593043cd-9a63-4043-8365-d97b79858319-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:25.596722 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.596688 5086 generic.go:356] "Generic (PLEG): container finished" podID="593043cd-9a63-4043-8365-d97b79858319" containerID="7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed" exitCode=0 Nov 03 11:21:25.596910 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.596742 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" Nov 03 11:21:25.596910 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.596779 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" event={"ID":"593043cd-9a63-4043-8365-d97b79858319","Type":"ContainerDied","Data":"7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed"} Nov 03 11:21:25.596910 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.596837 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45" event={"ID":"593043cd-9a63-4043-8365-d97b79858319","Type":"ContainerDied","Data":"b0d5e7bff6d9871e2f28843a4fd9fc3014cf418c29d02037438fec76aa756c34"} Nov 03 11:21:25.596910 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.596860 5086 scope.go:122] "RemoveContainer" containerID="7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed" Nov 03 11:21:25.610290 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.610267 5086 scope.go:122] "RemoveContainer" containerID="7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed" Nov 03 11:21:25.610664 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:21:25.610617 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed\": container with ID starting with 7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed not found: ID does not exist" containerID="7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed" Nov 03 11:21:25.610768 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.610676 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed"} err="failed to get container status \"7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed\": rpc error: code = NotFound desc = could not find container \"7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed\": container with ID starting with 7ad9d75b15eff8b204989cd54bbc4ebbae6a031b1b29c91fbbf99de4eef2c2ed not found: ID does not exist" Nov 03 11:21:25.624134 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.624101 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45"] Nov 03 11:21:25.629616 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:25.629589 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-8bb45"] Nov 03 11:21:26.174798 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:26.174754 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593043cd-9a63-4043-8365-d97b79858319" path="/var/lib/kubelet/pods/593043cd-9a63-4043-8365-d97b79858319/volumes" Nov 03 11:21:44.527254 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:44.527214 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn"] Nov 03 11:21:44.527775 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:44.527748 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" podUID="e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" containerName="maas-api" containerID="cri-o://a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4" gracePeriod=30 Nov 03 11:21:45.101622 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.101594 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:21:45.168641 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.168587 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-metrics-certs\") pod \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " Nov 03 11:21:45.168808 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.168723 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-kube-api-access-9ldfz\" (UniqueName: \"kubernetes.io/projected/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-kube-api-access-9ldfz\") pod \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " Nov 03 11:21:45.168808 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.168744 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-maas-api-tls\") pod \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " Nov 03 11:21:45.169042 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.169017 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-openshift-service-ca\") pod \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\" (UID: \"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738\") " Nov 03 11:21:45.169597 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.169560 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-openshift-service-ca" pod "e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" (UID: "e0f31b7a-c7b1-4ed9-a3d4-5274003f5738"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:21:45.169794 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.169777 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:45.173013 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.172987 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-metrics-certs" pod "e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" (UID: "e0f31b7a-c7b1-4ed9-a3d4-5274003f5738"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:21:45.173144 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.173116 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-maas-api-tls" pod "e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" (UID: "e0f31b7a-c7b1-4ed9-a3d4-5274003f5738"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:21:45.177752 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.177723 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-kube-api-access-9ldfz" pod "e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" (UID: "e0f31b7a-c7b1-4ed9-a3d4-5274003f5738"). InnerVolumeSpecName "kube-api-access-9ldfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:21:45.271224 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.271186 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:45.271224 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.271218 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ldfz\" (UniqueName: \"kubernetes.io/projected/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-kube-api-access-9ldfz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:45.271484 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.271235 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:21:45.828398 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.828361 5086 generic.go:356] "Generic (PLEG): container finished" podID="e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" containerID="a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4" exitCode=0 Nov 03 11:21:45.828848 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.828424 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" Nov 03 11:21:45.828848 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.828439 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" event={"ID":"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738","Type":"ContainerDied","Data":"a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4"} Nov 03 11:21:45.828848 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.828502 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn" event={"ID":"e0f31b7a-c7b1-4ed9-a3d4-5274003f5738","Type":"ContainerDied","Data":"fb251fa34f07ef16d5b0e7e5b5acb242587c65ac238718fcf1de4d099dca800f"} Nov 03 11:21:45.828848 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.828528 5086 scope.go:122] "RemoveContainer" containerID="a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4" Nov 03 11:21:45.839804 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.839789 5086 scope.go:122] "RemoveContainer" containerID="a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4" Nov 03 11:21:45.840036 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:21:45.840018 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4\": container with ID starting with a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4 not found: ID does not exist" containerID="a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4" Nov 03 11:21:45.840090 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.840044 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4"} err="failed to get container status \"a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4\": rpc error: code = NotFound desc = could not find container \"a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4\": container with ID starting with a8a6ec7c3211d103774d3e2fe82b37ffba32f32eaf9a013e6cf38f0da8bd7fa4 not found: ID does not exist" Nov 03 11:21:45.860937 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.860916 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn"] Nov 03 11:21:45.881680 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:45.881657 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-055d82-5876f65f88-9g7kn"] Nov 03 11:21:46.203691 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:21:46.203423 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" path="/var/lib/kubelet/pods/e0f31b7a-c7b1-4ed9-a3d4-5274003f5738/volumes" Nov 03 11:21:54.617371 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:21:54.605828 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:22:12.769658 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.769619 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7"] Nov 03 11:22:12.770739 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.770711 5086 cpu_manager.go:414] "containerMap: removing container" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" Nov 03 11:22:12.770739 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.770738 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" Nov 03 11:22:12.770917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.770778 5086 cpu_manager.go:414] "containerMap: removing container" podUID="e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" containerName="maas-api" Nov 03 11:22:12.770917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.770787 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" containerName="maas-api" Nov 03 11:22:12.770997 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.770949 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f31b7a-c7b1-4ed9-a3d4-5274003f5738" containerName="maas-api" Nov 03 11:22:12.770997 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.770973 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="593043cd-9a63-4043-8365-d97b79858319" containerName="istio-proxy" Nov 03 11:22:12.775907 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.775872 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:12.784593 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.784566 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7"] Nov 03 11:22:12.879320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.878790 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c0a748-openshift-default-dockercfg-q6b72\"" Nov 03 11:22:12.898725 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.898684 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:12.900046 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.900006 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:12.900360 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.900334 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:12.900599 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.900572 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:12.900779 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.900741 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:12.900900 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.900849 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:12.901015 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.900995 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26l7\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-kube-api-access-r26l7\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:12.901095 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.901037 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:12.901153 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:12.901098 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.002803 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.002768 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.002803 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.002808 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003065 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.002859 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003065 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.002929 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003065 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.002961 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003065 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.003001 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003313 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.003066 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26l7\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-kube-api-access-r26l7\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003313 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.003095 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003313 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.003150 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003313 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.003226 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003571 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.003348 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003571 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.003484 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003571 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.003543 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.003977 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.003954 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.005683 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.005659 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.005778 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.005765 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.012279 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.012253 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26l7\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-kube-api-access-r26l7\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.012580 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.012560 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.103663 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.103626 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:13.399520 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.399485 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7"] Nov 03 11:22:13.399915 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:22:13.399891 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2230d20d_db1c_40e8_abea_e5e6dee9dfb2.slice/crio-1b029f8f4f20af5106e15598b03ebd3e1a29078b706394410dd7648fe62e2bcb WatchSource:0}: Error finding container 1b029f8f4f20af5106e15598b03ebd3e1a29078b706394410dd7648fe62e2bcb: Status 404 returned error can't find the container with id 1b029f8f4f20af5106e15598b03ebd3e1a29078b706394410dd7648fe62e2bcb Nov 03 11:22:13.403100 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.403061 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:22:13.403215 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.403149 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:22:13.403215 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.403194 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:22:13.460206 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:13.460176 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" event={"ID":"2230d20d-db1c-40e8-abea-e5e6dee9dfb2","Type":"ContainerStarted","Data":"1b029f8f4f20af5106e15598b03ebd3e1a29078b706394410dd7648fe62e2bcb"} Nov 03 11:22:14.541402 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:14.534578 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" event={"ID":"2230d20d-db1c-40e8-abea-e5e6dee9dfb2","Type":"ContainerStarted","Data":"323ac24250647e9df1e1b0fa3a7dd3337b6a76820516f967b08bc2d26797d044"} Nov 03 11:22:14.558344 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:14.558251 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" podStartSLOduration=2.558206895 podStartE2EDuration="2.558206895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:22:14.555319699 +0000 UTC m=+3212.540190984" watchObservedRunningTime="2026-11-03 11:22:14.558206895 +0000 UTC m=+3212.543078178" Nov 03 11:22:15.103952 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:15.103923 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:15.106073 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:15.106043 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 11:22:15.106260 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:15.106099 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 11:22:16.104607 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:16.104575 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 11:22:16.104973 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:16.104628 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 11:22:17.104087 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:17.104054 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 11:22:17.104273 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:17.104110 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 11:22:18.104258 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:18.104221 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 11:22:18.104687 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:18.104272 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 11:22:19.104773 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:19.104729 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 11:22:19.105172 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:19.104790 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 11:22:20.104223 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:20.104185 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 11:22:20.104444 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:20.104239 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 11:22:21.031373 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.031338 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf"] Nov 03 11:22:21.031781 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.031562 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" podUID="5c395d47-3fca-4469-86b6-8ffd5f53234a" containerName="maas-api" containerID="cri-o://104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6" gracePeriod=30 Nov 03 11:22:21.109571 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.109538 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:21.109967 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.109945 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:21.110743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.110726 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:22:21.512031 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.512004 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:22:21.611046 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.611016 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls\") pod \"5c395d47-3fca-4469-86b6-8ffd5f53234a\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " Nov 03 11:22:21.611236 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.611073 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c395d47-3fca-4469-86b6-8ffd5f53234a-kube-api-access-z9cj5\" (UniqueName: \"kubernetes.io/projected/5c395d47-3fca-4469-86b6-8ffd5f53234a-kube-api-access-z9cj5\") pod \"5c395d47-3fca-4469-86b6-8ffd5f53234a\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " Nov 03 11:22:21.611236 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.611191 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-metrics-certs\") pod \"5c395d47-3fca-4469-86b6-8ffd5f53234a\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " Nov 03 11:22:21.611236 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.611226 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c395d47-3fca-4469-86b6-8ffd5f53234a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c395d47-3fca-4469-86b6-8ffd5f53234a-openshift-service-ca\") pod \"5c395d47-3fca-4469-86b6-8ffd5f53234a\" (UID: \"5c395d47-3fca-4469-86b6-8ffd5f53234a\") " Nov 03 11:22:21.611745 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.611716 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c395d47-3fca-4469-86b6-8ffd5f53234a-openshift-service-ca" pod "5c395d47-3fca-4469-86b6-8ffd5f53234a" (UID: "5c395d47-3fca-4469-86b6-8ffd5f53234a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:22:21.613873 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.613827 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-metrics-certs" pod "5c395d47-3fca-4469-86b6-8ffd5f53234a" (UID: "5c395d47-3fca-4469-86b6-8ffd5f53234a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:22:21.617548 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.614009 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls" pod "5c395d47-3fca-4469-86b6-8ffd5f53234a" (UID: "5c395d47-3fca-4469-86b6-8ffd5f53234a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:22:21.617548 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.614063 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c395d47-3fca-4469-86b6-8ffd5f53234a-kube-api-access-z9cj5" pod "5c395d47-3fca-4469-86b6-8ffd5f53234a" (UID: "5c395d47-3fca-4469-86b6-8ffd5f53234a"). InnerVolumeSpecName "kube-api-access-z9cj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:22:21.665196 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.665161 5086 generic.go:356] "Generic (PLEG): container finished" podID="5c395d47-3fca-4469-86b6-8ffd5f53234a" containerID="104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6" exitCode=0 Nov 03 11:22:21.665367 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.665310 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" event={"ID":"5c395d47-3fca-4469-86b6-8ffd5f53234a","Type":"ContainerDied","Data":"104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6"} Nov 03 11:22:21.665819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.665781 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" Nov 03 11:22:21.665929 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.665876 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" event={"ID":"5c395d47-3fca-4469-86b6-8ffd5f53234a","Type":"ContainerDied","Data":"daf439fc9b5fbba1e62c899c63da4d13e955bacdddb8d26f2b87c2ef15d45033"} Nov 03 11:22:21.665929 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.665926 5086 scope.go:122] "RemoveContainer" containerID="104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6" Nov 03 11:22:21.678129 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.678111 5086 scope.go:122] "RemoveContainer" containerID="104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6" Nov 03 11:22:21.678508 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:22:21.678487 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6\": container with ID starting with 104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6 not found: ID does not exist" containerID="104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6" Nov 03 11:22:21.678588 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.678515 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6"} err="failed to get container status \"104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6\": rpc error: code = NotFound desc = could not find container \"104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6\": container with ID starting with 104e4468fc09d18b3bd8c7f273770a7f4fc05dcf3748ec77a5e13c95014a78a6 not found: ID does not exist" Nov 03 11:22:21.690975 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.690948 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf"] Nov 03 11:22:21.694731 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.694707 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf"] Nov 03 11:22:21.713074 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.713041 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:21.713074 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.713071 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c395d47-3fca-4469-86b6-8ffd5f53234a-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:21.713233 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.713086 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c395d47-3fca-4469-86b6-8ffd5f53234a-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:21.713233 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:21.713100 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9cj5\" (UniqueName: \"kubernetes.io/projected/5c395d47-3fca-4469-86b6-8ffd5f53234a-kube-api-access-z9cj5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:22.172975 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:22.172940 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c395d47-3fca-4469-86b6-8ffd5f53234a" path="/var/lib/kubelet/pods/5c395d47-3fca-4469-86b6-8ffd5f53234a/volumes" Nov 03 11:22:23.995492 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:23.995454 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495"] Nov 03 11:22:23.996246 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:23.996231 5086 cpu_manager.go:414] "containerMap: removing container" podUID="5c395d47-3fca-4469-86b6-8ffd5f53234a" containerName="maas-api" Nov 03 11:22:23.996333 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:23.996248 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c395d47-3fca-4469-86b6-8ffd5f53234a" containerName="maas-api" Nov 03 11:22:23.996399 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:23.996389 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c395d47-3fca-4469-86b6-8ffd5f53234a" containerName="maas-api" Nov 03 11:22:24.001817 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.001792 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.011470 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.011448 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b13404-openshift-default-dockercfg-bxcqw\"" Nov 03 11:22:24.017733 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.016864 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495"] Nov 03 11:22:24.106315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.096899 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-credential-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.106315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.096957 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-data\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.106315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.097038 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cppv7\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-kube-api-access-cppv7\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.106315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.097142 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d745bb74-c06c-47b9-8b0e-388da32dd45b-istiod-ca-cert\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.106315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.097201 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-podinfo\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.106315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.097236 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.106315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.097258 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-token\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.106315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.097316 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-certs\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.106315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.097513 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-envoy\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.198569 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.198528 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-envoy\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.198754 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.198591 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-credential-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.198754 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.198628 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-data\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.198754 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.198693 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cppv7\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-kube-api-access-cppv7\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.198903 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.198786 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d745bb74-c06c-47b9-8b0e-388da32dd45b-istiod-ca-cert\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.198903 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.198846 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-podinfo\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.198903 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.198882 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.199056 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.198979 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-token\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.199056 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.199033 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-credential-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.199169 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.199059 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-certs\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.199169 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.199112 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-data\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.199278 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.199204 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.199499 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.199478 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-certs\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.199605 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.199586 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d745bb74-c06c-47b9-8b0e-388da32dd45b-istiod-ca-cert\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.201001 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.200981 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-envoy\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.201558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.201537 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-podinfo\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.206565 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.206539 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-token\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.206852 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.206831 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cppv7\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-kube-api-access-cppv7\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-jf495\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.275243 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:22:24.275160 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:22:24.326706 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.326673 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:24.440908 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.440863 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24376f-56bfdb9987-sbnwf" podUID="5c395d47-3fca-4469-86b6-8ffd5f53234a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.193:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 11:22:24.624519 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.624486 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495"] Nov 03 11:22:24.625247 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:22:24.625219 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd745bb74_c06c_47b9_8b0e_388da32dd45b.slice/crio-fe68b2e2d62e74d23bd06021ab18c4c9a9c19c67bb4ca0f6569049cb31ee3486 WatchSource:0}: Error finding container fe68b2e2d62e74d23bd06021ab18c4c9a9c19c67bb4ca0f6569049cb31ee3486: Status 404 returned error can't find the container with id fe68b2e2d62e74d23bd06021ab18c4c9a9c19c67bb4ca0f6569049cb31ee3486 Nov 03 11:22:24.629529 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.629489 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:22:24.629609 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.629581 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:22:24.630128 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.629634 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:22:24.717262 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:24.717225 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" event={"ID":"d745bb74-c06c-47b9-8b0e-388da32dd45b","Type":"ContainerStarted","Data":"fe68b2e2d62e74d23bd06021ab18c4c9a9c19c67bb4ca0f6569049cb31ee3486"} Nov 03 11:22:25.757784 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:25.757744 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" event={"ID":"d745bb74-c06c-47b9-8b0e-388da32dd45b","Type":"ContainerStarted","Data":"e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae"} Nov 03 11:22:25.778668 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:25.778617 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" podStartSLOduration=2.778603849 podStartE2EDuration="2.778603849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:22:25.775968289 +0000 UTC m=+3223.760839572" watchObservedRunningTime="2026-11-03 11:22:25.778603849 +0000 UTC m=+3223.763475131" Nov 03 11:22:26.339320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:26.338442 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:26.341474 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:26.341438 5086 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 11:22:26.341620 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:26.341508 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 11:22:27.340819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:27.332692 5086 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 11:22:27.340819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:27.332752 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 11:22:28.341349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:28.339431 5086 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 11:22:28.341349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:28.339493 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 11:22:29.335706 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:29.335667 5086 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 11:22:29.336009 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:29.335979 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 11:22:30.338318 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:30.332614 5086 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 11:22:30.338318 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:30.332677 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 11:22:31.519505 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:31.519463 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:31.520039 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:31.520013 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:31.520925 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:31.520901 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:22:38.379485 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.379448 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh"] Nov 03 11:22:38.379873 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.379702 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" podUID="eb691805-f85b-4e37-b201-0dd3137389b6" containerName="maas-api" containerID="cri-o://7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9" gracePeriod=30 Nov 03 11:22:38.783592 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.783561 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:22:38.891503 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.891468 5086 generic.go:356] "Generic (PLEG): container finished" podID="eb691805-f85b-4e37-b201-0dd3137389b6" containerID="7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9" exitCode=0 Nov 03 11:22:38.891671 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.891519 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" Nov 03 11:22:38.891671 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.891555 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" event={"ID":"eb691805-f85b-4e37-b201-0dd3137389b6","Type":"ContainerDied","Data":"7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9"} Nov 03 11:22:38.891671 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.891623 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh" event={"ID":"eb691805-f85b-4e37-b201-0dd3137389b6","Type":"ContainerDied","Data":"bf0de88f334a75d3fd1915efd3f122199290dde1703184ed33fa56796917e0a0"} Nov 03 11:22:38.891671 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.891651 5086 scope.go:122] "RemoveContainer" containerID="7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9" Nov 03 11:22:38.904329 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.904310 5086 scope.go:122] "RemoveContainer" containerID="7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9" Nov 03 11:22:38.904664 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:22:38.904638 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9\": container with ID starting with 7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9 not found: ID does not exist" containerID="7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9" Nov 03 11:22:38.904758 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.904673 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9"} err="failed to get container status \"7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9\": rpc error: code = NotFound desc = could not find container \"7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9\": container with ID starting with 7c7ff81dc9914378d2c88351b2f198f22aa1eb97b991b318bc9f95f49e081cc9 not found: ID does not exist" Nov 03 11:22:38.958821 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.958504 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-metrics-certs\") pod \"eb691805-f85b-4e37-b201-0dd3137389b6\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " Nov 03 11:22:38.958821 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.958690 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb691805-f85b-4e37-b201-0dd3137389b6-kube-api-access-l7thg\" (UniqueName: \"kubernetes.io/projected/eb691805-f85b-4e37-b201-0dd3137389b6-kube-api-access-l7thg\") pod \"eb691805-f85b-4e37-b201-0dd3137389b6\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " Nov 03 11:22:38.958821 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.958750 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls\") pod \"eb691805-f85b-4e37-b201-0dd3137389b6\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " Nov 03 11:22:38.959280 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.958947 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb691805-f85b-4e37-b201-0dd3137389b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb691805-f85b-4e37-b201-0dd3137389b6-openshift-service-ca\") pod \"eb691805-f85b-4e37-b201-0dd3137389b6\" (UID: \"eb691805-f85b-4e37-b201-0dd3137389b6\") " Nov 03 11:22:38.961373 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.959832 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb691805-f85b-4e37-b201-0dd3137389b6-openshift-service-ca" pod "eb691805-f85b-4e37-b201-0dd3137389b6" (UID: "eb691805-f85b-4e37-b201-0dd3137389b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:22:38.963592 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.963539 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-metrics-certs" pod "eb691805-f85b-4e37-b201-0dd3137389b6" (UID: "eb691805-f85b-4e37-b201-0dd3137389b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:22:38.963867 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.963844 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb691805-f85b-4e37-b201-0dd3137389b6-kube-api-access-l7thg" pod "eb691805-f85b-4e37-b201-0dd3137389b6" (UID: "eb691805-f85b-4e37-b201-0dd3137389b6"). InnerVolumeSpecName "kube-api-access-l7thg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:22:38.964584 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:38.964561 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls" pod "eb691805-f85b-4e37-b201-0dd3137389b6" (UID: "eb691805-f85b-4e37-b201-0dd3137389b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:22:39.060917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:39.060836 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7thg\" (UniqueName: \"kubernetes.io/projected/eb691805-f85b-4e37-b201-0dd3137389b6-kube-api-access-l7thg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:39.060917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:39.060867 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:39.060917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:39.060878 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb691805-f85b-4e37-b201-0dd3137389b6-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:39.060917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:39.060887 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb691805-f85b-4e37-b201-0dd3137389b6-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:39.224684 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:39.224650 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh"] Nov 03 11:22:39.230218 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:39.230193 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3eb43f-84456fbf96-cknwh"] Nov 03 11:22:40.172720 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:40.172687 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb691805-f85b-4e37-b201-0dd3137389b6" path="/var/lib/kubelet/pods/eb691805-f85b-4e37-b201-0dd3137389b6/volumes" Nov 03 11:22:48.522649 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:48.522612 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt"] Nov 03 11:22:48.523163 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:48.522805 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" containerID="cri-o://113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517" gracePeriod=30 Nov 03 11:22:52.706154 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:52.706116 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w"] Nov 03 11:22:52.709722 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:52.709686 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" podUID="a7979985-a6e6-4e9f-91f9-82d9c7940ed4" containerName="discovery" containerID="cri-o://dd12f08e1561d6fd376f5f5fbd25f17234c7b9fb7c991c6f23da75f404ebe60a" gracePeriod=30 Nov 03 11:22:53.078551 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.078513 5086 generic.go:356] "Generic (PLEG): container finished" podID="a7979985-a6e6-4e9f-91f9-82d9c7940ed4" containerID="dd12f08e1561d6fd376f5f5fbd25f17234c7b9fb7c991c6f23da75f404ebe60a" exitCode=0 Nov 03 11:22:53.079019 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.078584 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" event={"ID":"a7979985-a6e6-4e9f-91f9-82d9c7940ed4","Type":"ContainerDied","Data":"dd12f08e1561d6fd376f5f5fbd25f17234c7b9fb7c991c6f23da75f404ebe60a"} Nov 03 11:22:53.233564 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.233540 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:22:53.391057 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.391024 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-local-certs\") pod \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " Nov 03 11:22:53.391232 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.391103 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-dns-cert\") pod \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " Nov 03 11:22:53.391232 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.391130 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-cacerts\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-cacerts\") pod \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " Nov 03 11:22:53.391396 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.391374 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-kube-api-access-978s6\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-kube-api-access-978s6\") pod \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " Nov 03 11:22:53.391515 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.391496 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-token\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-token\") pod \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " Nov 03 11:22:53.391649 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.391542 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-ca-configmap\") pod \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " Nov 03 11:22:53.391649 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.391575 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-kubeconfig\") pod \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\" (UID: \"a7979985-a6e6-4e9f-91f9-82d9c7940ed4\") " Nov 03 11:22:53.392091 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.392063 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-ca-configmap" pod "a7979985-a6e6-4e9f-91f9-82d9c7940ed4" (UID: "a7979985-a6e6-4e9f-91f9-82d9c7940ed4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:22:53.392376 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.392347 5086 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-ca-configmap\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:53.393704 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.393669 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-local-certs" pod "a7979985-a6e6-4e9f-91f9-82d9c7940ed4" (UID: "a7979985-a6e6-4e9f-91f9-82d9c7940ed4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:22:53.393988 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.393964 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-cacerts" pod "a7979985-a6e6-4e9f-91f9-82d9c7940ed4" (UID: "a7979985-a6e6-4e9f-91f9-82d9c7940ed4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:22:53.394053 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.393975 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-dns-cert" pod "a7979985-a6e6-4e9f-91f9-82d9c7940ed4" (UID: "a7979985-a6e6-4e9f-91f9-82d9c7940ed4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:22:53.394224 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.394198 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-token" pod "a7979985-a6e6-4e9f-91f9-82d9c7940ed4" (UID: "a7979985-a6e6-4e9f-91f9-82d9c7940ed4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:22:53.394375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.394350 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-kube-api-access-978s6" pod "a7979985-a6e6-4e9f-91f9-82d9c7940ed4" (UID: "a7979985-a6e6-4e9f-91f9-82d9c7940ed4"). InnerVolumeSpecName "kube-api-access-978s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:22:53.394516 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.394501 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-kubeconfig" pod "a7979985-a6e6-4e9f-91f9-82d9c7940ed4" (UID: "a7979985-a6e6-4e9f-91f9-82d9c7940ed4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:22:53.493417 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.493386 5086 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-csr-dns-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:53.493417 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.493412 5086 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-cacerts\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:53.493614 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.493423 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-978s6\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-kube-api-access-978s6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:53.493614 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.493435 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:53.493614 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.493445 5086 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-istio-kubeconfig\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:53.493614 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.493453 5086 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7979985-a6e6-4e9f-91f9-82d9c7940ed4-local-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:53.810411 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:53.810386 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 11:22:54.000092 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000015 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-data\") pod \"ee3adb96-ea20-407b-bad4-7401e60bfd27\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " Nov 03 11:22:54.000092 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000076 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-kube-api-access-btlbj\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-kube-api-access-btlbj\") pod \"ee3adb96-ea20-407b-bad4-7401e60bfd27\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " Nov 03 11:22:54.000345 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000252 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-token\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-token\") pod \"ee3adb96-ea20-407b-bad4-7401e60bfd27\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " Nov 03 11:22:54.000427 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000343 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3adb96-ea20-407b-bad4-7401e60bfd27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3adb96-ea20-407b-bad4-7401e60bfd27-istiod-ca-cert\") pod \"ee3adb96-ea20-407b-bad4-7401e60bfd27\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " Nov 03 11:22:54.000427 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000349 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-data" pod "ee3adb96-ea20-407b-bad4-7401e60bfd27" (UID: "ee3adb96-ea20-407b-bad4-7401e60bfd27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:22:54.000427 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000391 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-socket\") pod \"ee3adb96-ea20-407b-bad4-7401e60bfd27\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " Nov 03 11:22:54.000603 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000531 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-certs\") pod \"ee3adb96-ea20-407b-bad4-7401e60bfd27\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " Nov 03 11:22:54.000603 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000563 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-envoy\") pod \"ee3adb96-ea20-407b-bad4-7401e60bfd27\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " Nov 03 11:22:54.000721 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000643 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-credential-socket\") pod \"ee3adb96-ea20-407b-bad4-7401e60bfd27\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " Nov 03 11:22:54.000721 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000673 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-podinfo\") pod \"ee3adb96-ea20-407b-bad4-7401e60bfd27\" (UID: \"ee3adb96-ea20-407b-bad4-7401e60bfd27\") " Nov 03 11:22:54.000721 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000702 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3adb96-ea20-407b-bad4-7401e60bfd27-istiod-ca-cert" pod "ee3adb96-ea20-407b-bad4-7401e60bfd27" (UID: "ee3adb96-ea20-407b-bad4-7401e60bfd27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:22:54.000890 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.000833 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-certs" pod "ee3adb96-ea20-407b-bad4-7401e60bfd27" (UID: "ee3adb96-ea20-407b-bad4-7401e60bfd27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:22:54.001048 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.001028 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-socket" pod "ee3adb96-ea20-407b-bad4-7401e60bfd27" (UID: "ee3adb96-ea20-407b-bad4-7401e60bfd27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:22:54.001142 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.001051 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-credential-socket" pod "ee3adb96-ea20-407b-bad4-7401e60bfd27" (UID: "ee3adb96-ea20-407b-bad4-7401e60bfd27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:22:54.001449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.001425 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:54.001545 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.001465 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:54.001545 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.001479 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:54.001545 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.001493 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3adb96-ea20-407b-bad4-7401e60bfd27-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:54.001545 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.001505 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:54.002950 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.002930 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-kube-api-access-btlbj" pod "ee3adb96-ea20-407b-bad4-7401e60bfd27" (UID: "ee3adb96-ea20-407b-bad4-7401e60bfd27"). InnerVolumeSpecName "kube-api-access-btlbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:22:54.003064 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.003047 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-token" pod "ee3adb96-ea20-407b-bad4-7401e60bfd27" (UID: "ee3adb96-ea20-407b-bad4-7401e60bfd27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:22:54.003213 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.003190 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-podinfo" pod "ee3adb96-ea20-407b-bad4-7401e60bfd27" (UID: "ee3adb96-ea20-407b-bad4-7401e60bfd27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:22:54.003213 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.003199 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-envoy" pod "ee3adb96-ea20-407b-bad4-7401e60bfd27" (UID: "ee3adb96-ea20-407b-bad4-7401e60bfd27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:22:54.088087 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.088062 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" Nov 03 11:22:54.088335 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.088097 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w" event={"ID":"a7979985-a6e6-4e9f-91f9-82d9c7940ed4","Type":"ContainerDied","Data":"53f26061a4e7a0539cd181033ec13354e4f9e16a40d188a377bb103cbfb41b2b"} Nov 03 11:22:54.088335 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.088177 5086 scope.go:122] "RemoveContainer" containerID="dd12f08e1561d6fd376f5f5fbd25f17234c7b9fb7c991c6f23da75f404ebe60a" Nov 03 11:22:54.089721 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.089694 5086 generic.go:356] "Generic (PLEG): container finished" podID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerID="113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517" exitCode=0 Nov 03 11:22:54.089836 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.089750 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" Nov 03 11:22:54.089836 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.089778 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" event={"ID":"ee3adb96-ea20-407b-bad4-7401e60bfd27","Type":"ContainerDied","Data":"113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517"} Nov 03 11:22:54.089924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.089836 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt" event={"ID":"ee3adb96-ea20-407b-bad4-7401e60bfd27","Type":"ContainerDied","Data":"3679d61e30d5641819af2941e62fd24b90e1bf5d4fe3317f50e51122ead32908"} Nov 03 11:22:54.102983 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.102958 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:54.103110 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.102985 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:54.103110 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.103001 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3adb96-ea20-407b-bad4-7401e60bfd27-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:54.103110 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.103016 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btlbj\" (UniqueName: \"kubernetes.io/projected/ee3adb96-ea20-407b-bad4-7401e60bfd27-kube-api-access-btlbj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:22:54.107718 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.107699 5086 scope.go:122] "RemoveContainer" containerID="113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517" Nov 03 11:22:54.119710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.119688 5086 scope.go:122] "RemoveContainer" containerID="113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517" Nov 03 11:22:54.119995 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:22:54.119976 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517\": container with ID starting with 113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517 not found: ID does not exist" containerID="113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517" Nov 03 11:22:54.120079 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.120004 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517"} err="failed to get container status \"113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517\": rpc error: code = NotFound desc = could not find container \"113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517\": container with ID starting with 113ddb83d37b2cb9929e7fa3b7f672319fa98ff7e832702efe893fac2d739517 not found: ID does not exist" Nov 03 11:22:54.122657 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.122633 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w"] Nov 03 11:22:54.126823 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.126802 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sb26w"] Nov 03 11:22:54.138542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.138520 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt"] Nov 03 11:22:54.141802 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.141785 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-fpbnt"] Nov 03 11:22:54.172900 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.172873 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7979985-a6e6-4e9f-91f9-82d9c7940ed4" path="/var/lib/kubelet/pods/a7979985-a6e6-4e9f-91f9-82d9c7940ed4/volumes" Nov 03 11:22:54.173440 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:54.173427 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" path="/var/lib/kubelet/pods/ee3adb96-ea20-407b-bad4-7401e60bfd27/volumes" Nov 03 11:22:54.275202 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:22:54.275115 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:22:57.763111 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.763073 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz"] Nov 03 11:22:57.764271 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.764243 5086 cpu_manager.go:414] "containerMap: removing container" podUID="a7979985-a6e6-4e9f-91f9-82d9c7940ed4" containerName="discovery" Nov 03 11:22:57.764271 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.764268 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7979985-a6e6-4e9f-91f9-82d9c7940ed4" containerName="discovery" Nov 03 11:22:57.764534 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.764338 5086 cpu_manager.go:414] "containerMap: removing container" podUID="eb691805-f85b-4e37-b201-0dd3137389b6" containerName="maas-api" Nov 03 11:22:57.764534 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.764348 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb691805-f85b-4e37-b201-0dd3137389b6" containerName="maas-api" Nov 03 11:22:57.764534 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.764364 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" Nov 03 11:22:57.764534 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.764371 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" Nov 03 11:22:57.764761 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.764568 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7979985-a6e6-4e9f-91f9-82d9c7940ed4" containerName="discovery" Nov 03 11:22:57.764761 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.764594 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb691805-f85b-4e37-b201-0dd3137389b6" containerName="maas-api" Nov 03 11:22:57.764761 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.764608 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3adb96-ea20-407b-bad4-7401e60bfd27" containerName="istio-proxy" Nov 03 11:22:57.770375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.770354 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.790521 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.790492 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz"] Nov 03 11:22:57.882385 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.882346 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.882564 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.882404 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.882564 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.882497 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.882661 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.882583 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qbc\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-kube-api-access-98qbc\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.882755 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.882733 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.882821 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.882793 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b025-0cf4-4501-95c3-e470d8089223-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.882944 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.882922 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.882996 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.882954 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.882996 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.882981 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.984601 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.984559 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.984601 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.984606 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.984829 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.984638 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.984894 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.984849 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.984955 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.984903 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.984955 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.984944 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.985069 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.985027 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qbc\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-kube-api-access-98qbc\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.985069 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.985053 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.985218 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.985199 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.985290 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.985229 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.985290 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.985265 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b025-0cf4-4501-95c3-e470d8089223-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.985430 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.985402 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.985543 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.985525 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.986010 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.985988 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b025-0cf4-4501-95c3-e470d8089223-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.987622 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.987599 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.987840 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.987824 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.994882 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.994862 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qbc\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-kube-api-access-98qbc\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:57.994966 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:57.994894 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:58.097028 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:58.096996 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:22:58.383796 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:58.383769 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz"] Nov 03 11:22:58.385713 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:22:58.385683 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc9b025_0cf4_4501_95c3_e470d8089223.slice/crio-df44fe126fb6d67f5e8e1dce609d63a021b0295abc1a7765f1840a8b36a73dff WatchSource:0}: Error finding container df44fe126fb6d67f5e8e1dce609d63a021b0295abc1a7765f1840a8b36a73dff: Status 404 returned error can't find the container with id df44fe126fb6d67f5e8e1dce609d63a021b0295abc1a7765f1840a8b36a73dff Nov 03 11:22:58.388874 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:58.388823 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:22:58.388992 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:58.388926 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:22:58.388992 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:58.388975 5086 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 11:22:59.182682 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:59.182189 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" event={"ID":"6bc9b025-0cf4-4501-95c3-e470d8089223","Type":"ContainerStarted","Data":"ab14ca1473d15c855deabb1cc10c16c74a1fc48148eec455a5a14dfb1c3223bd"} Nov 03 11:22:59.182682 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:59.182245 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" event={"ID":"6bc9b025-0cf4-4501-95c3-e470d8089223","Type":"ContainerStarted","Data":"df44fe126fb6d67f5e8e1dce609d63a021b0295abc1a7765f1840a8b36a73dff"} Nov 03 11:22:59.241361 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:22:59.240772 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" podStartSLOduration=2.24075028 podStartE2EDuration="2.24075028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:22:59.217827148 +0000 UTC m=+3257.202698429" watchObservedRunningTime="2026-11-03 11:22:59.24075028 +0000 UTC m=+3257.225621566" Nov 03 11:23:00.097474 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:00.097438 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:23:00.099830 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:00.099798 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 11:23:00.099969 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:00.099866 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 11:23:01.098084 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:01.098050 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 11:23:01.098498 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:01.098099 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 11:23:02.098329 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:02.098274 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 11:23:02.098809 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:02.098344 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 11:23:03.097864 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:03.097827 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 11:23:03.098073 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:03.097881 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 11:23:04.097599 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:04.097568 5086 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 11:23:04.098008 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:04.097629 5086 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 11:23:05.165658 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:05.165620 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:23:05.166647 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:05.166616 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:23:05.168348 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:05.168326 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:23:06.708341 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:06.707825 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l"] Nov 03 11:23:06.708341 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:06.708061 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" podUID="33d6f189-277a-47de-9410-2c52d9edea84" containerName="maas-api" containerID="cri-o://e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19" gracePeriod=30 Nov 03 11:23:07.273153 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.273126 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:23:07.329777 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.329736 5086 generic.go:356] "Generic (PLEG): container finished" podID="33d6f189-277a-47de-9410-2c52d9edea84" containerID="e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19" exitCode=0 Nov 03 11:23:07.329960 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.329840 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" Nov 03 11:23:07.329960 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.329873 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" event={"ID":"33d6f189-277a-47de-9410-2c52d9edea84","Type":"ContainerDied","Data":"e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19"} Nov 03 11:23:07.329960 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.329929 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l" event={"ID":"33d6f189-277a-47de-9410-2c52d9edea84","Type":"ContainerDied","Data":"963a8a48acecb41a3da4105261dfd30ad37978428be7c8ad26919ce42a18530e"} Nov 03 11:23:07.329960 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.329959 5086 scope.go:122] "RemoveContainer" containerID="e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19" Nov 03 11:23:07.332616 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.332590 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d6f189-277a-47de-9410-2c52d9edea84-kube-api-access-q5htj\" (UniqueName: \"kubernetes.io/projected/33d6f189-277a-47de-9410-2c52d9edea84-kube-api-access-q5htj\") pod \"33d6f189-277a-47de-9410-2c52d9edea84\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " Nov 03 11:23:07.332757 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.332629 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-metrics-certs\") pod \"33d6f189-277a-47de-9410-2c52d9edea84\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " Nov 03 11:23:07.332757 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.332691 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls\") pod \"33d6f189-277a-47de-9410-2c52d9edea84\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " Nov 03 11:23:07.332757 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.332734 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d6f189-277a-47de-9410-2c52d9edea84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d6f189-277a-47de-9410-2c52d9edea84-openshift-service-ca\") pod \"33d6f189-277a-47de-9410-2c52d9edea84\" (UID: \"33d6f189-277a-47de-9410-2c52d9edea84\") " Nov 03 11:23:07.334260 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.333465 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d6f189-277a-47de-9410-2c52d9edea84-openshift-service-ca" pod "33d6f189-277a-47de-9410-2c52d9edea84" (UID: "33d6f189-277a-47de-9410-2c52d9edea84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:23:07.335062 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.335042 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-metrics-certs" pod "33d6f189-277a-47de-9410-2c52d9edea84" (UID: "33d6f189-277a-47de-9410-2c52d9edea84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:23:07.335752 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.335719 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d6f189-277a-47de-9410-2c52d9edea84-kube-api-access-q5htj" pod "33d6f189-277a-47de-9410-2c52d9edea84" (UID: "33d6f189-277a-47de-9410-2c52d9edea84"). InnerVolumeSpecName "kube-api-access-q5htj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:23:07.336486 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.336461 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls" pod "33d6f189-277a-47de-9410-2c52d9edea84" (UID: "33d6f189-277a-47de-9410-2c52d9edea84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:23:07.373506 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.373475 5086 scope.go:122] "RemoveContainer" containerID="e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19" Nov 03 11:23:07.373912 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:23:07.373889 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19\": container with ID starting with e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19 not found: ID does not exist" containerID="e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19" Nov 03 11:23:07.373996 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.373920 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19"} err="failed to get container status \"e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19\": rpc error: code = NotFound desc = could not find container \"e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19\": container with ID starting with e88e02f7a1eeac0c8cbf8851bb9c6ab3180ed54e9d5aeb01d840b42109110a19 not found: ID does not exist" Nov 03 11:23:07.441328 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.438742 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d6f189-277a-47de-9410-2c52d9edea84-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:07.441328 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.438783 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5htj\" (UniqueName: \"kubernetes.io/projected/33d6f189-277a-47de-9410-2c52d9edea84-kube-api-access-q5htj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:07.441328 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.438798 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:07.441328 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.438810 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d6f189-277a-47de-9410-2c52d9edea84-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:07.654481 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.654440 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l"] Nov 03 11:23:07.659564 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:07.659540 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68aa46-565b57f6db-k6s9l"] Nov 03 11:23:08.179953 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:08.179910 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d6f189-277a-47de-9410-2c52d9edea84" path="/var/lib/kubelet/pods/33d6f189-277a-47de-9410-2c52d9edea84/volumes" Nov 03 11:23:20.007343 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:20.007281 5086 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" podUID="bb77e2dfdbc4a16f3bcede022ac475a7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Nov 03 11:23:24.275132 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:23:24.275049 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:23:39.564120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:39.564077 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7"] Nov 03 11:23:39.564935 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:39.564900 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" containerID="cri-o://323ac24250647e9df1e1b0fa3a7dd3337b6a76820516f967b08bc2d26797d044" gracePeriod=30 Nov 03 11:23:39.569237 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:39.569212 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz"] Nov 03 11:23:39.569443 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:39.569425 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerName="istio-proxy" containerID="cri-o://ab14ca1473d15c855deabb1cc10c16c74a1fc48148eec455a5a14dfb1c3223bd" gracePeriod=30 Nov 03 11:23:43.178727 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:43.178696 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:23:43.184389 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:43.184364 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:23:43.206080 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:43.206059 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:23:43.211588 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:43.211569 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:23:44.750150 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:23:44.750105 5086 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2230d20d_db1c_40e8_abea_e5e6dee9dfb2.slice/crio-323ac24250647e9df1e1b0fa3a7dd3337b6a76820516f967b08bc2d26797d044.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2230d20d_db1c_40e8_abea_e5e6dee9dfb2.slice/crio-conmon-323ac24250647e9df1e1b0fa3a7dd3337b6a76820516f967b08bc2d26797d044.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc9b025_0cf4_4501_95c3_e470d8089223.slice/crio-ab14ca1473d15c855deabb1cc10c16c74a1fc48148eec455a5a14dfb1c3223bd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc9b025_0cf4_4501_95c3_e470d8089223.slice/crio-conmon-ab14ca1473d15c855deabb1cc10c16c74a1fc48148eec455a5a14dfb1c3223bd.scope\": RecentStats: unable to find data in memory cache]" Nov 03 11:23:44.793805 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:44.793764 5086 generic.go:356] "Generic (PLEG): container finished" podID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerID="ab14ca1473d15c855deabb1cc10c16c74a1fc48148eec455a5a14dfb1c3223bd" exitCode=0 Nov 03 11:23:44.793974 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:44.793928 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" event={"ID":"6bc9b025-0cf4-4501-95c3-e470d8089223","Type":"ContainerDied","Data":"ab14ca1473d15c855deabb1cc10c16c74a1fc48148eec455a5a14dfb1c3223bd"} Nov 03 11:23:44.796383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:44.796351 5086 generic.go:356] "Generic (PLEG): container finished" podID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerID="323ac24250647e9df1e1b0fa3a7dd3337b6a76820516f967b08bc2d26797d044" exitCode=0 Nov 03 11:23:44.796548 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:44.796394 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" event={"ID":"2230d20d-db1c-40e8-abea-e5e6dee9dfb2","Type":"ContainerDied","Data":"323ac24250647e9df1e1b0fa3a7dd3337b6a76820516f967b08bc2d26797d044"} Nov 03 11:23:44.918493 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:44.918470 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:23:44.921167 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:44.921143 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:23:45.008061 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008025 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-kube-api-access-98qbc\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-kube-api-access-98qbc\") pod \"6bc9b025-0cf4-4501-95c3-e470d8089223\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " Nov 03 11:23:45.008281 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008093 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-socket\") pod \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " Nov 03 11:23:45.008281 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008169 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istiod-ca-cert\") pod \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " Nov 03 11:23:45.008281 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008203 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-token\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-token\") pod \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " Nov 03 11:23:45.008281 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008265 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-envoy\") pod \"6bc9b025-0cf4-4501-95c3-e470d8089223\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " Nov 03 11:23:45.008522 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008363 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-kube-api-access-r26l7\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-kube-api-access-r26l7\") pod \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " Nov 03 11:23:45.008522 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008386 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-socket\") pod \"6bc9b025-0cf4-4501-95c3-e470d8089223\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " Nov 03 11:23:45.008522 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008412 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-certs\") pod \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " Nov 03 11:23:45.008522 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008451 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-podinfo\") pod \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " Nov 03 11:23:45.008727 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008543 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-certs\") pod \"6bc9b025-0cf4-4501-95c3-e470d8089223\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " Nov 03 11:23:45.008727 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008576 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-podinfo\") pod \"6bc9b025-0cf4-4501-95c3-e470d8089223\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " Nov 03 11:23:45.008727 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008614 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc9b025-0cf4-4501-95c3-e470d8089223-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b025-0cf4-4501-95c3-e470d8089223-istiod-ca-cert\") pod \"6bc9b025-0cf4-4501-95c3-e470d8089223\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " Nov 03 11:23:45.008727 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008669 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-data\") pod \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " Nov 03 11:23:45.008727 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008706 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-token\") pod \"6bc9b025-0cf4-4501-95c3-e470d8089223\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " Nov 03 11:23:45.008974 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008743 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-data\") pod \"6bc9b025-0cf4-4501-95c3-e470d8089223\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " Nov 03 11:23:45.008974 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008781 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-credential-socket\") pod \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " Nov 03 11:23:45.008974 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008804 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-envoy\") pod \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\" (UID: \"2230d20d-db1c-40e8-abea-e5e6dee9dfb2\") " Nov 03 11:23:45.008974 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.008892 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-credential-socket\") pod \"6bc9b025-0cf4-4501-95c3-e470d8089223\" (UID: \"6bc9b025-0cf4-4501-95c3-e470d8089223\") " Nov 03 11:23:45.009330 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.009269 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-data" pod "2230d20d-db1c-40e8-abea-e5e6dee9dfb2" (UID: "2230d20d-db1c-40e8-abea-e5e6dee9dfb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.009569 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.009545 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-certs" pod "6bc9b025-0cf4-4501-95c3-e470d8089223" (UID: "6bc9b025-0cf4-4501-95c3-e470d8089223"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.009648 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.009611 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.011750 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.011718 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-socket" pod "2230d20d-db1c-40e8-abea-e5e6dee9dfb2" (UID: "2230d20d-db1c-40e8-abea-e5e6dee9dfb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.012104 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.012069 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istiod-ca-cert" pod "2230d20d-db1c-40e8-abea-e5e6dee9dfb2" (UID: "2230d20d-db1c-40e8-abea-e5e6dee9dfb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:23:45.012213 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.012075 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-token" pod "6bc9b025-0cf4-4501-95c3-e470d8089223" (UID: "6bc9b025-0cf4-4501-95c3-e470d8089223"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:23:45.012280 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.012207 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-token" pod "2230d20d-db1c-40e8-abea-e5e6dee9dfb2" (UID: "2230d20d-db1c-40e8-abea-e5e6dee9dfb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:23:45.012359 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.012329 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-data" pod "6bc9b025-0cf4-4501-95c3-e470d8089223" (UID: "6bc9b025-0cf4-4501-95c3-e470d8089223"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.012564 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.012541 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-credential-socket" pod "2230d20d-db1c-40e8-abea-e5e6dee9dfb2" (UID: "2230d20d-db1c-40e8-abea-e5e6dee9dfb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.012720 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.012701 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-credential-socket" pod "6bc9b025-0cf4-4501-95c3-e470d8089223" (UID: "6bc9b025-0cf4-4501-95c3-e470d8089223"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.012805 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.012716 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-socket" pod "6bc9b025-0cf4-4501-95c3-e470d8089223" (UID: "6bc9b025-0cf4-4501-95c3-e470d8089223"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.012923 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.012905 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-certs" pod "2230d20d-db1c-40e8-abea-e5e6dee9dfb2" (UID: "2230d20d-db1c-40e8-abea-e5e6dee9dfb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.012994 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.012974 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc9b025-0cf4-4501-95c3-e470d8089223-istiod-ca-cert" pod "6bc9b025-0cf4-4501-95c3-e470d8089223" (UID: "6bc9b025-0cf4-4501-95c3-e470d8089223"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:23:45.014219 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.014199 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-podinfo" pod "6bc9b025-0cf4-4501-95c3-e470d8089223" (UID: "6bc9b025-0cf4-4501-95c3-e470d8089223"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:23:45.014777 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.014752 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-envoy" pod "2230d20d-db1c-40e8-abea-e5e6dee9dfb2" (UID: "2230d20d-db1c-40e8-abea-e5e6dee9dfb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.015631 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.015608 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-podinfo" pod "2230d20d-db1c-40e8-abea-e5e6dee9dfb2" (UID: "2230d20d-db1c-40e8-abea-e5e6dee9dfb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:23:45.015729 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.015627 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-kube-api-access-r26l7" pod "2230d20d-db1c-40e8-abea-e5e6dee9dfb2" (UID: "2230d20d-db1c-40e8-abea-e5e6dee9dfb2"). InnerVolumeSpecName "kube-api-access-r26l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:23:45.015959 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.015934 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-kube-api-access-98qbc" pod "6bc9b025-0cf4-4501-95c3-e470d8089223" (UID: "6bc9b025-0cf4-4501-95c3-e470d8089223"). InnerVolumeSpecName "kube-api-access-98qbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:23:45.017233 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.017214 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-envoy" pod "6bc9b025-0cf4-4501-95c3-e470d8089223" (UID: "6bc9b025-0cf4-4501-95c3-e470d8089223"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:23:45.110479 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110450 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110479 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110477 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110479 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110486 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110494 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b025-0cf4-4501-95c3-e470d8089223-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110503 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110511 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110519 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110527 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110537 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110546 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98qbc\" (UniqueName: \"kubernetes.io/projected/6bc9b025-0cf4-4501-95c3-e470d8089223-kube-api-access-98qbc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110556 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110564 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110572 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110580 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110588 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26l7\" (UniqueName: \"kubernetes.io/projected/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-kube-api-access-r26l7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110596 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b025-0cf4-4501-95c3-e470d8089223-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.110696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.110603 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2230d20d-db1c-40e8-abea-e5e6dee9dfb2-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:23:45.806260 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.806230 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" Nov 03 11:23:45.806260 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.806244 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz" event={"ID":"6bc9b025-0cf4-4501-95c3-e470d8089223","Type":"ContainerDied","Data":"df44fe126fb6d67f5e8e1dce609d63a021b0295abc1a7765f1840a8b36a73dff"} Nov 03 11:23:45.806833 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.806320 5086 scope.go:122] "RemoveContainer" containerID="ab14ca1473d15c855deabb1cc10c16c74a1fc48148eec455a5a14dfb1c3223bd" Nov 03 11:23:45.808129 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.808095 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" Nov 03 11:23:45.808129 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.808089 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7" event={"ID":"2230d20d-db1c-40e8-abea-e5e6dee9dfb2","Type":"ContainerDied","Data":"1b029f8f4f20af5106e15598b03ebd3e1a29078b706394410dd7648fe62e2bcb"} Nov 03 11:23:45.819897 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.819876 5086 scope.go:122] "RemoveContainer" containerID="323ac24250647e9df1e1b0fa3a7dd3337b6a76820516f967b08bc2d26797d044" Nov 03 11:23:45.832953 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.832932 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz"] Nov 03 11:23:45.839270 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.839250 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-slblz"] Nov 03 11:23:45.850029 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.849950 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7"] Nov 03 11:23:45.854017 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:45.853997 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-wrjh7"] Nov 03 11:23:46.174070 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:46.173997 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" path="/var/lib/kubelet/pods/2230d20d-db1c-40e8-abea-e5e6dee9dfb2/volumes" Nov 03 11:23:46.174558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:23:46.174543 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" path="/var/lib/kubelet/pods/6bc9b025-0cf4-4501-95c3-e470d8089223/volumes" Nov 03 11:23:54.279563 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:23:54.277368 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:24:04.571461 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.571426 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6"] Nov 03 11:24:04.572405 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.572384 5086 cpu_manager.go:414] "containerMap: removing container" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" Nov 03 11:24:04.572405 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.572405 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" Nov 03 11:24:04.572585 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.572443 5086 cpu_manager.go:414] "containerMap: removing container" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerName="istio-proxy" Nov 03 11:24:04.572585 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.572451 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerName="istio-proxy" Nov 03 11:24:04.572585 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.572499 5086 cpu_manager.go:414] "containerMap: removing container" podUID="33d6f189-277a-47de-9410-2c52d9edea84" containerName="maas-api" Nov 03 11:24:04.572585 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.572507 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d6f189-277a-47de-9410-2c52d9edea84" containerName="maas-api" Nov 03 11:24:04.572790 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.572680 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d6f189-277a-47de-9410-2c52d9edea84" containerName="maas-api" Nov 03 11:24:04.572790 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.572700 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="2230d20d-db1c-40e8-abea-e5e6dee9dfb2" containerName="istio-proxy" Nov 03 11:24:04.572790 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.572716 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc9b025-0cf4-4501-95c3-e470d8089223" containerName="istio-proxy" Nov 03 11:24:04.578514 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.578487 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.591162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.591066 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-049d3b4c-serving-cert\"" Nov 03 11:24:04.592801 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.592585 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6"] Nov 03 11:24:04.731152 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.731117 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-metrics-certs\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.731152 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.731153 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lpm\" (UniqueName: \"kubernetes.io/projected/8d41b16f-8d5a-476d-bdd1-b181795591da-kube-api-access-64lpm\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.731445 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.731343 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-maas-api-tls\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.731518 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.731501 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d41b16f-8d5a-476d-bdd1-b181795591da-openshift-service-ca\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.833476 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.833378 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-metrics-certs\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.833476 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.833442 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lpm\" (UniqueName: \"kubernetes.io/projected/8d41b16f-8d5a-476d-bdd1-b181795591da-kube-api-access-64lpm\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.833728 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.833570 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-maas-api-tls\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.833790 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.833733 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d41b16f-8d5a-476d-bdd1-b181795591da-openshift-service-ca\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.834453 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.834427 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d41b16f-8d5a-476d-bdd1-b181795591da-openshift-service-ca\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.835993 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.835969 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-metrics-certs\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.836084 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.836034 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-maas-api-tls\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.841695 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.841670 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lpm\" (UniqueName: \"kubernetes.io/projected/8d41b16f-8d5a-476d-bdd1-b181795591da-kube-api-access-64lpm\") pod \"maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:04.911616 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:04.911581 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:05.197162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:05.197130 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6"] Nov 03 11:24:05.197931 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:24:05.197907 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d41b16f_8d5a_476d_bdd1_b181795591da.slice/crio-743374a9d6fae8101c13cd7420eff71cbb7fe36ac80dc1508e41441d6364d9ca WatchSource:0}: Error finding container 743374a9d6fae8101c13cd7420eff71cbb7fe36ac80dc1508e41441d6364d9ca: Status 404 returned error can't find the container with id 743374a9d6fae8101c13cd7420eff71cbb7fe36ac80dc1508e41441d6364d9ca Nov 03 11:24:05.199570 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:05.199551 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 11:24:05.959669 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:05.959629 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" event={"ID":"8d41b16f-8d5a-476d-bdd1-b181795591da","Type":"ContainerStarted","Data":"cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093"} Nov 03 11:24:05.960142 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:05.959692 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" event={"ID":"8d41b16f-8d5a-476d-bdd1-b181795591da","Type":"ContainerStarted","Data":"743374a9d6fae8101c13cd7420eff71cbb7fe36ac80dc1508e41441d6364d9ca"} Nov 03 11:24:05.960142 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:05.959721 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:05.975996 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:05.975938 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" podStartSLOduration=1.371893795 podStartE2EDuration="1.975920017s" totalImagesPullingTime="604.026222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:24:05.975690449 +0000 UTC m=+3323.960561733" watchObservedRunningTime="2026-11-03 11:24:05.975920017 +0000 UTC m=+3323.960791300" Nov 03 11:24:11.980796 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:11.980751 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" podUID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.199:8443/health\": dial tcp 10.132.0.199:8443: connect: connection refused" Nov 03 11:24:16.048821 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:16.048783 5086 generic.go:356] "Generic (PLEG): container finished" podID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerID="cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093" exitCode=1 Nov 03 11:24:16.049346 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:16.048860 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" event={"ID":"8d41b16f-8d5a-476d-bdd1-b181795591da","Type":"ContainerDied","Data":"cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093"} Nov 03 11:24:16.049456 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:16.049438 5086 scope.go:122] "RemoveContainer" containerID="cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093" Nov 03 11:24:17.058025 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:17.057981 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" event={"ID":"8d41b16f-8d5a-476d-bdd1-b181795591da","Type":"ContainerStarted","Data":"334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3"} Nov 03 11:24:17.058466 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:17.058236 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:23.106370 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:23.106339 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:24:24.275908 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:24:24.275866 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:24:35.572349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:35.572280 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g"] Nov 03 11:24:35.572886 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:35.572561 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" containerID="cri-o://3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6" gracePeriod=30 Nov 03 11:24:40.378776 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:40.378742 5086 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 11:24:40.379166 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:40.378806 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 11:24:40.893675 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:40.893649 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 11:24:41.014955 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.014874 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-certs\") pod \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " Nov 03 11:24:41.014955 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.014928 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-socket\") pod \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " Nov 03 11:24:41.015201 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015014 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istiod-ca-cert\") pod \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " Nov 03 11:24:41.015201 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015041 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-credential-socket\") pod \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " Nov 03 11:24:41.015201 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015074 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-envoy\") pod \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " Nov 03 11:24:41.015201 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015132 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-kube-api-access-tfdqt\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-kube-api-access-tfdqt\") pod \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " Nov 03 11:24:41.015201 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015196 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-token\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-token\") pod \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " Nov 03 11:24:41.015510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015217 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-certs" pod "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" (UID: "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:24:41.015510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015219 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-socket" pod "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" (UID: "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:24:41.015510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015259 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-podinfo\") pod \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " Nov 03 11:24:41.015510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015430 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-credential-socket" pod "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" (UID: "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:24:41.015510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015450 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-data\") pod \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\" (UID: \"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee\") " Nov 03 11:24:41.015510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015500 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istiod-ca-cert" pod "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" (UID: "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:24:41.015806 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.015652 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-data" pod "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" (UID: "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:24:41.016215 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.016193 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:24:41.016324 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.016221 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:24:41.016324 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.016237 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:24:41.016324 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.016253 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:24:41.016324 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.016268 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:24:41.017590 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.017568 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-podinfo" pod "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" (UID: "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:24:41.017680 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.017608 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-kube-api-access-tfdqt" pod "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" (UID: "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee"). InnerVolumeSpecName "kube-api-access-tfdqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:24:41.017882 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.017853 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-envoy" pod "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" (UID: "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:24:41.017928 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.017878 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-token" pod "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" (UID: "c1980046-fc71-4a79-aef8-bc8b5a6dd3ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:24:41.117151 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.117113 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:24:41.117151 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.117142 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfdqt\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-kube-api-access-tfdqt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:24:41.117151 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.117153 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:24:41.117412 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.117164 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:24:41.342070 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.342030 5086 generic.go:356] "Generic (PLEG): container finished" podID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerID="3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6" exitCode=0 Nov 03 11:24:41.342249 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.342086 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" Nov 03 11:24:41.342249 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.342113 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" event={"ID":"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee","Type":"ContainerDied","Data":"3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6"} Nov 03 11:24:41.342249 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.342174 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g" event={"ID":"c1980046-fc71-4a79-aef8-bc8b5a6dd3ee","Type":"ContainerDied","Data":"d82e317585b2d8f879fcfd3a3e2943b298f62cc9164732cb8bdf5293325885b9"} Nov 03 11:24:41.342249 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.342197 5086 scope.go:122] "RemoveContainer" containerID="3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6" Nov 03 11:24:41.360493 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.354378 5086 scope.go:122] "RemoveContainer" containerID="3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6" Nov 03 11:24:41.360665 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:24:41.360623 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6\": container with ID starting with 3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6 not found: ID does not exist" containerID="3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6" Nov 03 11:24:41.360838 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.360813 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6"} err="failed to get container status \"3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6\": rpc error: code = NotFound desc = could not find container \"3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6\": container with ID starting with 3f9049968e0dff2e31cb103f4f146539a3c974ef84f6a37a913acf567673f9c6 not found: ID does not exist" Nov 03 11:24:41.369862 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.369830 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g"] Nov 03 11:24:41.374336 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:41.374316 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6h57g"] Nov 03 11:24:42.172234 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:42.172203 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" path="/var/lib/kubelet/pods/c1980046-fc71-4a79-aef8-bc8b5a6dd3ee/volumes" Nov 03 11:24:46.786441 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.785500 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m"] Nov 03 11:24:46.786923 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.786872 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" Nov 03 11:24:46.786923 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.786893 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" Nov 03 11:24:46.796343 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.787119 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1980046-fc71-4a79-aef8-bc8b5a6dd3ee" containerName="istio-proxy" Nov 03 11:24:46.801851 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.801004 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:46.839334 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.818715 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce5555\"/\"kube-root-ca.crt\"" Nov 03 11:24:46.839334 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.819484 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce5555\"/\"openshift-service-ca.crt\"" Nov 03 11:24:46.839334 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.825743 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m"] Nov 03 11:24:46.839334 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.826575 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce5555\"/\"default-dockercfg-xcnf9\"" Nov 03 11:24:46.839334 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.826724 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce5555\"/\"rate-test-model-ce5555-kserve-self-signed-certs\"" Nov 03 11:24:46.938320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.937711 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-dshm\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:46.938320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.937907 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-tmp-dir\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:46.938320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.937951 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-model-cache\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:46.938320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.938006 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea433f-d697-460c-8e67-32b03caa1440-tls-certs\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:46.938320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.938093 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdtx\" (UniqueName: \"kubernetes.io/projected/b4ea433f-d697-460c-8e67-32b03caa1440-kube-api-access-hfdtx\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:46.938320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:46.938132 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-home\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.052375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.048597 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-tmp-dir\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.052375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.048651 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-model-cache\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.052375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.048706 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea433f-d697-460c-8e67-32b03caa1440-tls-certs\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.052375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.049415 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-tmp-dir\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.052375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.049632 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-model-cache\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.052375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.049688 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdtx\" (UniqueName: \"kubernetes.io/projected/b4ea433f-d697-460c-8e67-32b03caa1440-kube-api-access-hfdtx\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.052375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.049770 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-home\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.052375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.049908 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-dshm\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.052375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.051830 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-home\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.057185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.055851 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea433f-d697-460c-8e67-32b03caa1440-tls-certs\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.057185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.056993 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-dshm\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.058527 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.058113 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdtx\" (UniqueName: \"kubernetes.io/projected/b4ea433f-d697-460c-8e67-32b03caa1440-kube-api-access-hfdtx\") pod \"rate-test-model-ce5555-kserve-5f6f467f7-7s96m\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.148861 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.148617 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:47.514168 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:47.514138 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m"] Nov 03 11:24:48.492918 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:48.492552 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" event={"ID":"b4ea433f-d697-460c-8e67-32b03caa1440","Type":"ContainerStarted","Data":"511ce26580d4ea6ce4344066c78979225b8db97a8fcd119d88f3c19513b01b65"} Nov 03 11:24:48.492918 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:48.492605 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" event={"ID":"b4ea433f-d697-460c-8e67-32b03caa1440","Type":"ContainerStarted","Data":"bf5794035e1b14d53c8df8b90b475fbe231a7830ccb65abc0f59ee4383fbf896"} Nov 03 11:24:48.512501 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:48.512449 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" podStartSLOduration=2.512435933 podStartE2EDuration="2.512435933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:24:48.51120618 +0000 UTC m=+3366.496077464" watchObservedRunningTime="2026-11-03 11:24:48.512435933 +0000 UTC m=+3366.497307216" Nov 03 11:24:49.162324 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:49.161412 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:49.236318 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:49.224380 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5p4bj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 11:24:49.236318 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:49.224458 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5p4bj" podUID="b7804ed8-ba49-4936-99f0-f02253369475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 11:24:49.258315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:49.243392 5086 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lrmsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 11:24:49.258315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:49.243460 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lrmsr" podUID="99bbba2f-874c-4246-9ec3-e92e117a4159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 11:24:54.275693 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:24:54.275613 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:24:57.160480 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:57.160440 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:57.176061 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:57.176025 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:24:57.566508 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:24:57.566483 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:25:05.314386 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.314348 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k"] Nov 03 11:25:05.320457 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.320435 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.331873 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.331843 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"rate-test-model-836589-kserve-self-signed-certs\"" Nov 03 11:25:05.332569 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.332157 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"kube-root-ca.crt\"" Nov 03 11:25:05.332688 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.332248 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"openshift-service-ca.crt\"" Nov 03 11:25:05.332688 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.332499 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"default-dockercfg-mclmp\"" Nov 03 11:25:05.340754 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.340727 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k"] Nov 03 11:25:05.379797 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.379767 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-home\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.379969 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.379838 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrj7t\" (UniqueName: \"kubernetes.io/projected/390485cb-9990-4cb9-8a8e-38e57f80b2cf-kube-api-access-xrj7t\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.380020 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.379976 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tls-certs\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.380120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.380096 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-model-cache\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.380233 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.380159 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-dshm\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.380508 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.380281 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tmp-dir\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.481605 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.481566 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tmp-dir\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.481776 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.481699 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-home\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.481776 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.481756 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrj7t\" (UniqueName: \"kubernetes.io/projected/390485cb-9990-4cb9-8a8e-38e57f80b2cf-kube-api-access-xrj7t\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.481951 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.481862 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tls-certs\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.482055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.481949 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-model-cache\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.482055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.481972 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tmp-dir\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.482055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.482008 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-dshm\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.482263 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.482224 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-model-cache\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.482590 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.482327 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-home\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.484423 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.484400 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-dshm\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.485388 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.484844 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tls-certs\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.500176 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.500139 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrj7t\" (UniqueName: \"kubernetes.io/projected/390485cb-9990-4cb9-8a8e-38e57f80b2cf-kube-api-access-xrj7t\") pod \"rate-test-model-836589-kserve-7d6f5579cc-dbp9k\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.642662 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.642596 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:05.932771 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:05.932678 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k"] Nov 03 11:25:05.940856 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:25:05.940823 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod390485cb_9990_4cb9_8a8e_38e57f80b2cf.slice/crio-127df9c39658aaff90d7ee475e58dee94ded310faf8acf100e9a9014e117312e WatchSource:0}: Error finding container 127df9c39658aaff90d7ee475e58dee94ded310faf8acf100e9a9014e117312e: Status 404 returned error can't find the container with id 127df9c39658aaff90d7ee475e58dee94ded310faf8acf100e9a9014e117312e Nov 03 11:25:06.627925 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:06.627882 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" event={"ID":"390485cb-9990-4cb9-8a8e-38e57f80b2cf","Type":"ContainerStarted","Data":"8708997c5f6fac6e1edc2cf42d51f67a99a0065eac5a3fe5d5ebc54e87281781"} Nov 03 11:25:06.628463 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:06.627941 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" event={"ID":"390485cb-9990-4cb9-8a8e-38e57f80b2cf","Type":"ContainerStarted","Data":"127df9c39658aaff90d7ee475e58dee94ded310faf8acf100e9a9014e117312e"} Nov 03 11:25:06.643261 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:06.643233 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:06.651887 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:06.651832 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" podStartSLOduration=1.651817838 podStartE2EDuration="1.651817838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:25:06.648550897 +0000 UTC m=+3384.633422181" watchObservedRunningTime="2026-11-03 11:25:06.651817838 +0000 UTC m=+3384.636689120" Nov 03 11:25:15.643242 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:15.643210 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:15.655948 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:15.655922 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:15.702633 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:25:15.702605 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:25:24.301888 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:25:24.301849 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:25:54.278353 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:25:54.278269 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:26:10.537424 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:10.537379 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d"] Nov 03 11:26:10.538030 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:10.537645 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" podUID="a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" containerName="maas-api" containerID="cri-o://7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2" gracePeriod=30 Nov 03 11:26:11.067867 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.067838 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:26:11.089761 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.089679 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-maas-api-tls\") pod \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " Nov 03 11:26:11.089920 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.089888 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-kube-api-access-w6t7k\" (UniqueName: \"kubernetes.io/projected/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-kube-api-access-w6t7k\") pod \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " Nov 03 11:26:11.091169 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.089995 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-openshift-service-ca\") pod \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " Nov 03 11:26:11.091169 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.090039 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-metrics-certs\") pod \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\" (UID: \"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c\") " Nov 03 11:26:11.091169 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.090692 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-openshift-service-ca" pod "a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" (UID: "a9c1b8dd-1f29-47a3-8a2b-a2f56526131c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:26:11.091169 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.091138 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:11.093123 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.092939 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-kube-api-access-w6t7k" pod "a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" (UID: "a9c1b8dd-1f29-47a3-8a2b-a2f56526131c"). InnerVolumeSpecName "kube-api-access-w6t7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:26:11.093228 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.093175 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-maas-api-tls" pod "a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" (UID: "a9c1b8dd-1f29-47a3-8a2b-a2f56526131c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:26:11.093889 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.093867 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-metrics-certs" pod "a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" (UID: "a9c1b8dd-1f29-47a3-8a2b-a2f56526131c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:26:11.166415 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.166379 5086 generic.go:356] "Generic (PLEG): container finished" podID="a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" containerID="7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2" exitCode=0 Nov 03 11:26:11.166586 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.166438 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" Nov 03 11:26:11.166586 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.166480 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" event={"ID":"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c","Type":"ContainerDied","Data":"7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2"} Nov 03 11:26:11.166586 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.166528 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d" event={"ID":"a9c1b8dd-1f29-47a3-8a2b-a2f56526131c","Type":"ContainerDied","Data":"79eb27fd2dad89241036d87963cf000c63f44f12606bff998ac7e2e0ac89347e"} Nov 03 11:26:11.166586 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.166552 5086 scope.go:122] "RemoveContainer" containerID="7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2" Nov 03 11:26:11.179213 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.179197 5086 scope.go:122] "RemoveContainer" containerID="7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2" Nov 03 11:26:11.179513 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:26:11.179493 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2\": container with ID starting with 7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2 not found: ID does not exist" containerID="7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2" Nov 03 11:26:11.179594 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.179523 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2"} err="failed to get container status \"7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2\": rpc error: code = NotFound desc = could not find container \"7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2\": container with ID starting with 7bafeb827b983941ee3f1c0b5707cdf816f4638841cf259721aa150c623538c2 not found: ID does not exist" Nov 03 11:26:11.195170 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.195135 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d"] Nov 03 11:26:11.197068 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.197032 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6t7k\" (UniqueName: \"kubernetes.io/projected/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-kube-api-access-w6t7k\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:11.197068 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.197063 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:11.197326 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.197090 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:11.206608 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:11.206582 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b13404-75884796cd-v4q2d"] Nov 03 11:26:12.174612 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:12.174576 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" path="/var/lib/kubelet/pods/a9c1b8dd-1f29-47a3-8a2b-a2f56526131c/volumes" Nov 03 11:26:15.350121 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.350084 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6"] Nov 03 11:26:15.350652 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.350354 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" podUID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerName="maas-api" containerID="cri-o://334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3" gracePeriod=30 Nov 03 11:26:15.764862 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.764832 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:26:15.879435 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.879390 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d41b16f-8d5a-476d-bdd1-b181795591da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d41b16f-8d5a-476d-bdd1-b181795591da-openshift-service-ca\") pod \"8d41b16f-8d5a-476d-bdd1-b181795591da\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " Nov 03 11:26:15.879924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.879813 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d41b16f-8d5a-476d-bdd1-b181795591da-openshift-service-ca" pod "8d41b16f-8d5a-476d-bdd1-b181795591da" (UID: "8d41b16f-8d5a-476d-bdd1-b181795591da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:26:15.880040 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.880014 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-metrics-certs\") pod \"8d41b16f-8d5a-476d-bdd1-b181795591da\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " Nov 03 11:26:15.880499 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.880476 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d41b16f-8d5a-476d-bdd1-b181795591da-kube-api-access-64lpm\" (UniqueName: \"kubernetes.io/projected/8d41b16f-8d5a-476d-bdd1-b181795591da-kube-api-access-64lpm\") pod \"8d41b16f-8d5a-476d-bdd1-b181795591da\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " Nov 03 11:26:15.880889 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.880866 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-maas-api-tls\") pod \"8d41b16f-8d5a-476d-bdd1-b181795591da\" (UID: \"8d41b16f-8d5a-476d-bdd1-b181795591da\") " Nov 03 11:26:15.882069 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.882042 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d41b16f-8d5a-476d-bdd1-b181795591da-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:15.883281 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.883251 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-maas-api-tls" pod "8d41b16f-8d5a-476d-bdd1-b181795591da" (UID: "8d41b16f-8d5a-476d-bdd1-b181795591da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:26:15.883632 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.883608 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d41b16f-8d5a-476d-bdd1-b181795591da-kube-api-access-64lpm" pod "8d41b16f-8d5a-476d-bdd1-b181795591da" (UID: "8d41b16f-8d5a-476d-bdd1-b181795591da"). InnerVolumeSpecName "kube-api-access-64lpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:26:15.888077 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.888048 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-metrics-certs" pod "8d41b16f-8d5a-476d-bdd1-b181795591da" (UID: "8d41b16f-8d5a-476d-bdd1-b181795591da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:26:15.983409 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.983272 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:15.983409 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.983342 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d41b16f-8d5a-476d-bdd1-b181795591da-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:15.983409 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:15.983354 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lpm\" (UniqueName: \"kubernetes.io/projected/8d41b16f-8d5a-476d-bdd1-b181795591da-kube-api-access-64lpm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:16.231318 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.231246 5086 generic.go:356] "Generic (PLEG): container finished" podID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerID="334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3" exitCode=0 Nov 03 11:26:16.231888 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.231853 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" event={"ID":"8d41b16f-8d5a-476d-bdd1-b181795591da","Type":"ContainerDied","Data":"334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3"} Nov 03 11:26:16.232220 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.232167 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" event={"ID":"8d41b16f-8d5a-476d-bdd1-b181795591da","Type":"ContainerDied","Data":"743374a9d6fae8101c13cd7420eff71cbb7fe36ac80dc1508e41441d6364d9ca"} Nov 03 11:26:16.232413 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.232397 5086 scope.go:122] "RemoveContainer" containerID="334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3" Nov 03 11:26:16.232766 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.232743 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6" Nov 03 11:26:16.262013 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.261977 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6"] Nov 03 11:26:16.263339 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.263315 5086 scope.go:122] "RemoveContainer" containerID="cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093" Nov 03 11:26:16.265987 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.265969 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049d3b4c-79bd85d7b7-457w6"] Nov 03 11:26:16.273614 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.273589 5086 scope.go:122] "RemoveContainer" containerID="334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3" Nov 03 11:26:16.273886 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:26:16.273867 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3\": container with ID starting with 334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3 not found: ID does not exist" containerID="334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3" Nov 03 11:26:16.273941 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.273895 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3"} err="failed to get container status \"334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3\": rpc error: code = NotFound desc = could not find container \"334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3\": container with ID starting with 334f5d97212241c86186983f37d6c5b619374665c9f3442594050dea70e6e8d3 not found: ID does not exist" Nov 03 11:26:16.273941 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.273912 5086 scope.go:122] "RemoveContainer" containerID="cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093" Nov 03 11:26:16.274121 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:26:16.274103 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093\": container with ID starting with cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093 not found: ID does not exist" containerID="cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093" Nov 03 11:26:16.274164 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:16.274126 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093"} err="failed to get container status \"cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093\": rpc error: code = NotFound desc = could not find container \"cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093\": container with ID starting with cc1e0ae2d83bdb3c52d1d1099cd2478623b4546ecd361faf917b96c9dbcc3093 not found: ID does not exist" Nov 03 11:26:18.174923 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:18.174876 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d41b16f-8d5a-476d-bdd1-b181795591da" path="/var/lib/kubelet/pods/8d41b16f-8d5a-476d-bdd1-b181795591da/volumes" Nov 03 11:26:18.297486 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:18.297445 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495"] Nov 03 11:26:18.298013 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:18.297973 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerName="istio-proxy" containerID="cri-o://e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae" gracePeriod=30 Nov 03 11:26:23.634440 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.634412 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:26:23.736391 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.736355 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-podinfo\") pod \"d745bb74-c06c-47b9-8b0e-388da32dd45b\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " Nov 03 11:26:23.736587 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.736421 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-token\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-token\") pod \"d745bb74-c06c-47b9-8b0e-388da32dd45b\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " Nov 03 11:26:23.736661 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.736624 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-credential-socket\") pod \"d745bb74-c06c-47b9-8b0e-388da32dd45b\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " Nov 03 11:26:23.736723 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.736667 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-data\") pod \"d745bb74-c06c-47b9-8b0e-388da32dd45b\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " Nov 03 11:26:23.736787 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.736742 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-envoy\") pod \"d745bb74-c06c-47b9-8b0e-388da32dd45b\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " Nov 03 11:26:23.736851 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.736782 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-certs\") pod \"d745bb74-c06c-47b9-8b0e-388da32dd45b\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " Nov 03 11:26:23.736910 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.736845 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-socket\") pod \"d745bb74-c06c-47b9-8b0e-388da32dd45b\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " Nov 03 11:26:23.736981 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.736923 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-credential-socket" pod "d745bb74-c06c-47b9-8b0e-388da32dd45b" (UID: "d745bb74-c06c-47b9-8b0e-388da32dd45b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:26:23.737034 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.736994 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d745bb74-c06c-47b9-8b0e-388da32dd45b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d745bb74-c06c-47b9-8b0e-388da32dd45b-istiod-ca-cert\") pod \"d745bb74-c06c-47b9-8b0e-388da32dd45b\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " Nov 03 11:26:23.737034 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.737026 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-kube-api-access-cppv7\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-kube-api-access-cppv7\") pod \"d745bb74-c06c-47b9-8b0e-388da32dd45b\" (UID: \"d745bb74-c06c-47b9-8b0e-388da32dd45b\") " Nov 03 11:26:23.737139 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.737047 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-data" pod "d745bb74-c06c-47b9-8b0e-388da32dd45b" (UID: "d745bb74-c06c-47b9-8b0e-388da32dd45b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:26:23.737924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.737241 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-socket" pod "d745bb74-c06c-47b9-8b0e-388da32dd45b" (UID: "d745bb74-c06c-47b9-8b0e-388da32dd45b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:26:23.737924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.737573 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:23.737924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.737587 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:23.737924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.737596 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:23.737924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.737868 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-certs" pod "d745bb74-c06c-47b9-8b0e-388da32dd45b" (UID: "d745bb74-c06c-47b9-8b0e-388da32dd45b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:26:23.738681 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.738643 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d745bb74-c06c-47b9-8b0e-388da32dd45b-istiod-ca-cert" pod "d745bb74-c06c-47b9-8b0e-388da32dd45b" (UID: "d745bb74-c06c-47b9-8b0e-388da32dd45b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:26:23.739389 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.739339 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-token" pod "d745bb74-c06c-47b9-8b0e-388da32dd45b" (UID: "d745bb74-c06c-47b9-8b0e-388da32dd45b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:26:23.739533 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.739506 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-envoy" pod "d745bb74-c06c-47b9-8b0e-388da32dd45b" (UID: "d745bb74-c06c-47b9-8b0e-388da32dd45b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:26:23.739857 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.739832 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-podinfo" pod "d745bb74-c06c-47b9-8b0e-388da32dd45b" (UID: "d745bb74-c06c-47b9-8b0e-388da32dd45b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:26:23.739979 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.739953 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-kube-api-access-cppv7" pod "d745bb74-c06c-47b9-8b0e-388da32dd45b" (UID: "d745bb74-c06c-47b9-8b0e-388da32dd45b"). InnerVolumeSpecName "kube-api-access-cppv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:26:23.838659 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.838623 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:23.838659 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.838653 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:23.838659 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.838663 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d745bb74-c06c-47b9-8b0e-388da32dd45b-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:23.838914 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.838672 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d745bb74-c06c-47b9-8b0e-388da32dd45b-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:23.838914 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.838683 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cppv7\" (UniqueName: \"kubernetes.io/projected/d745bb74-c06c-47b9-8b0e-388da32dd45b-kube-api-access-cppv7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:23.838914 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:23.838692 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d745bb74-c06c-47b9-8b0e-388da32dd45b-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:26:24.275744 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:26:24.275704 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:26:24.292270 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:24.292237 5086 generic.go:356] "Generic (PLEG): container finished" podID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerID="e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae" exitCode=0 Nov 03 11:26:24.292470 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:24.292286 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" event={"ID":"d745bb74-c06c-47b9-8b0e-388da32dd45b","Type":"ContainerDied","Data":"e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae"} Nov 03 11:26:24.292470 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:24.292290 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" Nov 03 11:26:24.292470 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:24.292355 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495" event={"ID":"d745bb74-c06c-47b9-8b0e-388da32dd45b","Type":"ContainerDied","Data":"fe68b2e2d62e74d23bd06021ab18c4c9a9c19c67bb4ca0f6569049cb31ee3486"} Nov 03 11:26:24.292470 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:24.292385 5086 scope.go:122] "RemoveContainer" containerID="e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae" Nov 03 11:26:24.303488 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:24.303465 5086 scope.go:122] "RemoveContainer" containerID="e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae" Nov 03 11:26:24.303770 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:26:24.303752 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae\": container with ID starting with e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae not found: ID does not exist" containerID="e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae" Nov 03 11:26:24.303814 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:24.303779 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae"} err="failed to get container status \"e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae\": rpc error: code = NotFound desc = could not find container \"e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae\": container with ID starting with e69f8dc1c79abc4cdc03618a1f926c582a023ed4a12cb8d7cde5df12d5c81eae not found: ID does not exist" Nov 03 11:26:24.312437 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:24.312412 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495"] Nov 03 11:26:24.318386 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:24.318361 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-jf495"] Nov 03 11:26:26.214842 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:26:26.213972 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" path="/var/lib/kubelet/pods/d745bb74-c06c-47b9-8b0e-388da32dd45b/volumes" Nov 03 11:26:54.278677 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:26:54.278635 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:27:24.276708 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:27:24.276670 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:27:29.571481 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.571446 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt"] Nov 03 11:27:29.572258 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.572207 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" podUID="14889afa-093e-40af-93d5-28a889c72d6c" containerName="maas-api" containerID="cri-o://f8741e04c611319868cc010259d60f3319e13d6aa0cc0151e8bb0e4d84ea4229" gracePeriod=30 Nov 03 11:27:29.805355 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.805317 5086 generic.go:356] "Generic (PLEG): container finished" podID="14889afa-093e-40af-93d5-28a889c72d6c" containerID="f8741e04c611319868cc010259d60f3319e13d6aa0cc0151e8bb0e4d84ea4229" exitCode=0 Nov 03 11:27:29.805531 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.805385 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" event={"ID":"14889afa-093e-40af-93d5-28a889c72d6c","Type":"ContainerDied","Data":"f8741e04c611319868cc010259d60f3319e13d6aa0cc0151e8bb0e4d84ea4229"} Nov 03 11:27:29.867790 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.867762 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:27:29.906195 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.905985 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-maas-api-tls\") pod \"14889afa-093e-40af-93d5-28a889c72d6c\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " Nov 03 11:27:29.906195 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.906187 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-metrics-certs\") pod \"14889afa-093e-40af-93d5-28a889c72d6c\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " Nov 03 11:27:29.928576 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.906260 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14889afa-093e-40af-93d5-28a889c72d6c-kube-api-access-9rz57\" (UniqueName: \"kubernetes.io/projected/14889afa-093e-40af-93d5-28a889c72d6c-kube-api-access-9rz57\") pod \"14889afa-093e-40af-93d5-28a889c72d6c\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " Nov 03 11:27:29.928576 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.906369 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14889afa-093e-40af-93d5-28a889c72d6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14889afa-093e-40af-93d5-28a889c72d6c-openshift-service-ca\") pod \"14889afa-093e-40af-93d5-28a889c72d6c\" (UID: \"14889afa-093e-40af-93d5-28a889c72d6c\") " Nov 03 11:27:29.928576 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.907203 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14889afa-093e-40af-93d5-28a889c72d6c-openshift-service-ca" pod "14889afa-093e-40af-93d5-28a889c72d6c" (UID: "14889afa-093e-40af-93d5-28a889c72d6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:27:29.928576 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.907767 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14889afa-093e-40af-93d5-28a889c72d6c-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:29.969364 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.963619 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14889afa-093e-40af-93d5-28a889c72d6c-kube-api-access-9rz57" pod "14889afa-093e-40af-93d5-28a889c72d6c" (UID: "14889afa-093e-40af-93d5-28a889c72d6c"). InnerVolumeSpecName "kube-api-access-9rz57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:27:29.981319 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:29.969964 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-maas-api-tls" pod "14889afa-093e-40af-93d5-28a889c72d6c" (UID: "14889afa-093e-40af-93d5-28a889c72d6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:27:30.014428 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:30.011568 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rz57\" (UniqueName: \"kubernetes.io/projected/14889afa-093e-40af-93d5-28a889c72d6c-kube-api-access-9rz57\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:30.014428 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:30.011632 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:30.019434 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:30.018226 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-metrics-certs" pod "14889afa-093e-40af-93d5-28a889c72d6c" (UID: "14889afa-093e-40af-93d5-28a889c72d6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:27:30.113114 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:30.113026 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14889afa-093e-40af-93d5-28a889c72d6c-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:30.814993 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:30.814959 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" event={"ID":"14889afa-093e-40af-93d5-28a889c72d6c","Type":"ContainerDied","Data":"b09d2e8618599cfbccaf4d782bf9e06986d34de846d2a55c0e39e004bd0c6c38"} Nov 03 11:27:30.814993 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:30.814993 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt" Nov 03 11:27:30.815509 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:30.815027 5086 scope.go:122] "RemoveContainer" containerID="f8741e04c611319868cc010259d60f3319e13d6aa0cc0151e8bb0e4d84ea4229" Nov 03 11:27:30.836582 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:30.836555 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt"] Nov 03 11:27:30.842280 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:30.842255 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d9f2ce-8496dcc89b-sk6jt"] Nov 03 11:27:32.172480 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:32.172444 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14889afa-093e-40af-93d5-28a889c72d6c" path="/var/lib/kubelet/pods/14889afa-093e-40af-93d5-28a889c72d6c/volumes" Nov 03 11:27:37.942792 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:37.942750 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg"] Nov 03 11:27:37.943210 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:37.943023 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" containerID="cri-o://5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1" gracePeriod=30 Nov 03 11:27:41.183351 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:41.183309 5086 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 11:27:41.183848 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:41.183383 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 11:27:43.245877 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.245440 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:27:43.422808 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.422777 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-credential-socket\") pod \"3aa80780-52f3-4910-b141-4fc80639cc6e\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " Nov 03 11:27:43.423009 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.422832 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-token\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-token\") pod \"3aa80780-52f3-4910-b141-4fc80639cc6e\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " Nov 03 11:27:43.423009 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.422862 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa80780-52f3-4910-b141-4fc80639cc6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa80780-52f3-4910-b141-4fc80639cc6e-istiod-ca-cert\") pod \"3aa80780-52f3-4910-b141-4fc80639cc6e\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " Nov 03 11:27:43.423009 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.422895 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-podinfo\") pod \"3aa80780-52f3-4910-b141-4fc80639cc6e\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " Nov 03 11:27:43.423009 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.422915 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-socket\") pod \"3aa80780-52f3-4910-b141-4fc80639cc6e\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " Nov 03 11:27:43.423223 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423092 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-certs\") pod \"3aa80780-52f3-4910-b141-4fc80639cc6e\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " Nov 03 11:27:43.423223 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423128 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-credential-socket" pod "3aa80780-52f3-4910-b141-4fc80639cc6e" (UID: "3aa80780-52f3-4910-b141-4fc80639cc6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:27:43.423223 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423179 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-kube-api-access-c7czv\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-kube-api-access-c7czv\") pod \"3aa80780-52f3-4910-b141-4fc80639cc6e\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " Nov 03 11:27:43.423223 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423219 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-data\") pod \"3aa80780-52f3-4910-b141-4fc80639cc6e\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " Nov 03 11:27:43.423519 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423248 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-envoy\") pod \"3aa80780-52f3-4910-b141-4fc80639cc6e\" (UID: \"3aa80780-52f3-4910-b141-4fc80639cc6e\") " Nov 03 11:27:43.423519 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423350 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-socket" pod "3aa80780-52f3-4910-b141-4fc80639cc6e" (UID: "3aa80780-52f3-4910-b141-4fc80639cc6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:27:43.423519 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423461 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa80780-52f3-4910-b141-4fc80639cc6e-istiod-ca-cert" pod "3aa80780-52f3-4910-b141-4fc80639cc6e" (UID: "3aa80780-52f3-4910-b141-4fc80639cc6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:27:43.423697 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423573 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-data" pod "3aa80780-52f3-4910-b141-4fc80639cc6e" (UID: "3aa80780-52f3-4910-b141-4fc80639cc6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:27:43.423758 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423733 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-certs" pod "3aa80780-52f3-4910-b141-4fc80639cc6e" (UID: "3aa80780-52f3-4910-b141-4fc80639cc6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:27:43.424000 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.423972 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:43.424000 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.424001 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:43.424262 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.424016 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa80780-52f3-4910-b141-4fc80639cc6e-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:43.424262 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.424030 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:43.424262 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.424044 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:43.425380 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.425356 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-podinfo" pod "3aa80780-52f3-4910-b141-4fc80639cc6e" (UID: "3aa80780-52f3-4910-b141-4fc80639cc6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:27:43.425473 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.425445 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-token" pod "3aa80780-52f3-4910-b141-4fc80639cc6e" (UID: "3aa80780-52f3-4910-b141-4fc80639cc6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:27:43.425532 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.425491 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-kube-api-access-c7czv" pod "3aa80780-52f3-4910-b141-4fc80639cc6e" (UID: "3aa80780-52f3-4910-b141-4fc80639cc6e"). InnerVolumeSpecName "kube-api-access-c7czv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:27:43.425532 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.425491 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-envoy" pod "3aa80780-52f3-4910-b141-4fc80639cc6e" (UID: "3aa80780-52f3-4910-b141-4fc80639cc6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:27:43.525899 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.525809 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:43.525899 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.525840 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:43.525899 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.525853 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7czv\" (UniqueName: \"kubernetes.io/projected/3aa80780-52f3-4910-b141-4fc80639cc6e-kube-api-access-c7czv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:43.525899 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.525868 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa80780-52f3-4910-b141-4fc80639cc6e-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:43.908198 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.908162 5086 generic.go:356] "Generic (PLEG): container finished" podID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerID="5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1" exitCode=0 Nov 03 11:27:43.908411 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.908205 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" Nov 03 11:27:43.908411 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.908239 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" event={"ID":"3aa80780-52f3-4910-b141-4fc80639cc6e","Type":"ContainerDied","Data":"5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1"} Nov 03 11:27:43.908411 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.908318 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg" event={"ID":"3aa80780-52f3-4910-b141-4fc80639cc6e","Type":"ContainerDied","Data":"8c31a3742b7aaf6d022202ac4122b449ce77432506ac8e6b3a8bfad9f016f650"} Nov 03 11:27:43.908411 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.908348 5086 scope.go:122] "RemoveContainer" containerID="5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1" Nov 03 11:27:43.919733 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.919713 5086 scope.go:122] "RemoveContainer" containerID="5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1" Nov 03 11:27:43.920022 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:27:43.920003 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1\": container with ID starting with 5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1 not found: ID does not exist" containerID="5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1" Nov 03 11:27:43.920095 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.920031 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1"} err="failed to get container status \"5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1\": rpc error: code = NotFound desc = could not find container \"5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1\": container with ID starting with 5f504ddde6a7d6b86e906b1d9e6f0b86c25b9b9536f3e4a7fd195d41452dd3f1 not found: ID does not exist" Nov 03 11:27:43.937050 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.937022 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg"] Nov 03 11:27:43.941114 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:43.941094 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-7jdrg"] Nov 03 11:27:44.223247 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:44.223164 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" path="/var/lib/kubelet/pods/3aa80780-52f3-4910-b141-4fc80639cc6e/volumes" Nov 03 11:27:52.681051 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:52.681012 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx"] Nov 03 11:27:52.681947 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:52.681911 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" podUID="eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" containerName="maas-api" containerID="cri-o://cc891f7044dd0e78f582541d870f01af59163a9a1b434c86621f286495486c27" gracePeriod=30 Nov 03 11:27:53.004835 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.004730 5086 generic.go:356] "Generic (PLEG): container finished" podID="eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" containerID="cc891f7044dd0e78f582541d870f01af59163a9a1b434c86621f286495486c27" exitCode=0 Nov 03 11:27:53.005043 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.004980 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" event={"ID":"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2","Type":"ContainerDied","Data":"cc891f7044dd0e78f582541d870f01af59163a9a1b434c86621f286495486c27"} Nov 03 11:27:53.069595 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.069564 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 11:27:53.123646 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.123616 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-kube-api-access-vtxkw\" (UniqueName: \"kubernetes.io/projected/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-kube-api-access-vtxkw\") pod \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " Nov 03 11:27:53.123847 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.123705 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-metrics-certs\") pod \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " Nov 03 11:27:53.123847 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.123742 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-openshift-service-ca\") pod \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " Nov 03 11:27:53.123847 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.123774 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-maas-api-tls\") pod \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\" (UID: \"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2\") " Nov 03 11:27:53.124202 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.124157 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-openshift-service-ca" pod "eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" (UID: "eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:27:53.124725 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.124697 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:53.126624 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.126584 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-metrics-certs" pod "eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" (UID: "eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:27:53.126830 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.126804 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-maas-api-tls" pod "eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" (UID: "eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:27:53.129202 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.129179 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-kube-api-access-vtxkw" pod "eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" (UID: "eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2"). InnerVolumeSpecName "kube-api-access-vtxkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:27:53.225674 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.225642 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtxkw\" (UniqueName: \"kubernetes.io/projected/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-kube-api-access-vtxkw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:53.225674 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.225679 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:53.225905 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:53.225694 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:27:54.014243 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:54.014211 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" Nov 03 11:27:54.014675 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:54.014239 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx" event={"ID":"eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2","Type":"ContainerDied","Data":"59d0a939662f32f4d30ba8b712706e0d9acadffc83c2f985d36b4b59518f17d4"} Nov 03 11:27:54.014675 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:54.014319 5086 scope.go:122] "RemoveContainer" containerID="cc891f7044dd0e78f582541d870f01af59163a9a1b434c86621f286495486c27" Nov 03 11:27:54.040518 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:54.040493 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx"] Nov 03 11:27:54.046130 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:54.046107 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-47a8a4-644db6d9c5-nt2fx"] Nov 03 11:27:54.175181 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:27:54.175149 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" path="/var/lib/kubelet/pods/eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2/volumes" Nov 03 11:27:54.274508 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:27:54.274420 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:28:00.901508 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:00.901471 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj"] Nov 03 11:28:00.902060 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:00.901737 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" containerID="cri-o://f3c0dc1645d1e20b8841897eeff95a38af7224090e2d612f25ff0bf83c4feee7" gracePeriod=30 Nov 03 11:28:06.110900 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.110866 5086 generic.go:356] "Generic (PLEG): container finished" podID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerID="f3c0dc1645d1e20b8841897eeff95a38af7224090e2d612f25ff0bf83c4feee7" exitCode=0 Nov 03 11:28:06.111326 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.110975 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" event={"ID":"faeb4aef-28a7-447c-a5f9-7992a41382f9","Type":"ContainerDied","Data":"f3c0dc1645d1e20b8841897eeff95a38af7224090e2d612f25ff0bf83c4feee7"} Nov 03 11:28:06.193417 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.193357 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:28:06.220484 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.220453 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faeb4aef-28a7-447c-a5f9-7992a41382f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeb4aef-28a7-447c-a5f9-7992a41382f9-istiod-ca-cert\") pod \"faeb4aef-28a7-447c-a5f9-7992a41382f9\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " Nov 03 11:28:06.220739 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.220718 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-socket\") pod \"faeb4aef-28a7-447c-a5f9-7992a41382f9\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " Nov 03 11:28:06.220889 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.220863 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faeb4aef-28a7-447c-a5f9-7992a41382f9-istiod-ca-cert" pod "faeb4aef-28a7-447c-a5f9-7992a41382f9" (UID: "faeb4aef-28a7-447c-a5f9-7992a41382f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:28:06.221055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.220905 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-token\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-token\") pod \"faeb4aef-28a7-447c-a5f9-7992a41382f9\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " Nov 03 11:28:06.221279 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.221260 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-credential-socket\") pod \"faeb4aef-28a7-447c-a5f9-7992a41382f9\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " Nov 03 11:28:06.221395 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.221168 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-socket" pod "faeb4aef-28a7-447c-a5f9-7992a41382f9" (UID: "faeb4aef-28a7-447c-a5f9-7992a41382f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:28:06.221484 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.221391 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-envoy\") pod \"faeb4aef-28a7-447c-a5f9-7992a41382f9\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " Nov 03 11:28:06.221622 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.221602 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-credential-socket" pod "faeb4aef-28a7-447c-a5f9-7992a41382f9" (UID: "faeb4aef-28a7-447c-a5f9-7992a41382f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:28:06.225088 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.225064 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-envoy" pod "faeb4aef-28a7-447c-a5f9-7992a41382f9" (UID: "faeb4aef-28a7-447c-a5f9-7992a41382f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:28:06.226401 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.226378 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-data\") pod \"faeb4aef-28a7-447c-a5f9-7992a41382f9\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " Nov 03 11:28:06.226506 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.226439 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-podinfo\") pod \"faeb4aef-28a7-447c-a5f9-7992a41382f9\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " Nov 03 11:28:06.226506 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.226469 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-certs\") pod \"faeb4aef-28a7-447c-a5f9-7992a41382f9\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " Nov 03 11:28:06.226630 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.226561 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-kube-api-access-h5qkx\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-kube-api-access-h5qkx\") pod \"faeb4aef-28a7-447c-a5f9-7992a41382f9\" (UID: \"faeb4aef-28a7-447c-a5f9-7992a41382f9\") " Nov 03 11:28:06.227205 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.227171 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:06.227205 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.227191 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeb4aef-28a7-447c-a5f9-7992a41382f9-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:06.230489 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.227205 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:06.230489 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.227220 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:06.230489 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.227652 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-data" pod "faeb4aef-28a7-447c-a5f9-7992a41382f9" (UID: "faeb4aef-28a7-447c-a5f9-7992a41382f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:28:06.230489 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.228100 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-certs" pod "faeb4aef-28a7-447c-a5f9-7992a41382f9" (UID: "faeb4aef-28a7-447c-a5f9-7992a41382f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:28:06.230711 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.230609 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-token" pod "faeb4aef-28a7-447c-a5f9-7992a41382f9" (UID: "faeb4aef-28a7-447c-a5f9-7992a41382f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:28:06.230796 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.230772 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-podinfo" pod "faeb4aef-28a7-447c-a5f9-7992a41382f9" (UID: "faeb4aef-28a7-447c-a5f9-7992a41382f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:28:06.231010 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.230985 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-kube-api-access-h5qkx" pod "faeb4aef-28a7-447c-a5f9-7992a41382f9" (UID: "faeb4aef-28a7-447c-a5f9-7992a41382f9"). InnerVolumeSpecName "kube-api-access-h5qkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:28:06.328677 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.328646 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:06.328820 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.328701 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:06.328820 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.328739 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeb4aef-28a7-447c-a5f9-7992a41382f9-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:06.328820 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.328751 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeb4aef-28a7-447c-a5f9-7992a41382f9-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:06.328820 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:06.328765 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5qkx\" (UniqueName: \"kubernetes.io/projected/faeb4aef-28a7-447c-a5f9-7992a41382f9-kube-api-access-h5qkx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:07.118719 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:07.118691 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" Nov 03 11:28:07.119151 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:07.118716 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj" event={"ID":"faeb4aef-28a7-447c-a5f9-7992a41382f9","Type":"ContainerDied","Data":"26a8ea6538450b21b823a060d70e2cd3af3a5d73a71e84597a2025aaaef74cbc"} Nov 03 11:28:07.119151 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:07.118781 5086 scope.go:122] "RemoveContainer" containerID="f3c0dc1645d1e20b8841897eeff95a38af7224090e2d612f25ff0bf83c4feee7" Nov 03 11:28:07.163613 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:07.163573 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj"] Nov 03 11:28:07.168376 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:07.168346 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-2lbgj"] Nov 03 11:28:08.171388 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.171351 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" path="/var/lib/kubelet/pods/faeb4aef-28a7-447c-a5f9-7992a41382f9/volumes" Nov 03 11:28:08.550794 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.550714 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b"] Nov 03 11:28:08.551003 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.550958 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" podUID="d1852843-24f8-4260-acb4-e4c38514a646" containerName="maas-api" containerID="cri-o://a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0" gracePeriod=30 Nov 03 11:28:08.836582 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.836548 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:28:08.874286 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.873884 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-metrics-certs\") pod \"d1852843-24f8-4260-acb4-e4c38514a646\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " Nov 03 11:28:08.875356 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.874627 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls\") pod \"d1852843-24f8-4260-acb4-e4c38514a646\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " Nov 03 11:28:08.875356 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.874683 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1852843-24f8-4260-acb4-e4c38514a646-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1852843-24f8-4260-acb4-e4c38514a646-openshift-service-ca\") pod \"d1852843-24f8-4260-acb4-e4c38514a646\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " Nov 03 11:28:08.875356 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.874772 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1852843-24f8-4260-acb4-e4c38514a646-kube-api-access-h2wst\" (UniqueName: \"kubernetes.io/projected/d1852843-24f8-4260-acb4-e4c38514a646-kube-api-access-h2wst\") pod \"d1852843-24f8-4260-acb4-e4c38514a646\" (UID: \"d1852843-24f8-4260-acb4-e4c38514a646\") " Nov 03 11:28:08.876486 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.876460 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1852843-24f8-4260-acb4-e4c38514a646-openshift-service-ca" pod "d1852843-24f8-4260-acb4-e4c38514a646" (UID: "d1852843-24f8-4260-acb4-e4c38514a646"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:28:08.876739 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.876711 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-metrics-certs" pod "d1852843-24f8-4260-acb4-e4c38514a646" (UID: "d1852843-24f8-4260-acb4-e4c38514a646"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:28:08.902361 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.899276 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls" pod "d1852843-24f8-4260-acb4-e4c38514a646" (UID: "d1852843-24f8-4260-acb4-e4c38514a646"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:28:08.925911 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.923921 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1852843-24f8-4260-acb4-e4c38514a646-kube-api-access-h2wst" pod "d1852843-24f8-4260-acb4-e4c38514a646" (UID: "d1852843-24f8-4260-acb4-e4c38514a646"). InnerVolumeSpecName "kube-api-access-h2wst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:28:08.977585 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.977541 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2wst\" (UniqueName: \"kubernetes.io/projected/d1852843-24f8-4260-acb4-e4c38514a646-kube-api-access-h2wst\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:08.977585 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.977577 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:08.977585 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.977592 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1852843-24f8-4260-acb4-e4c38514a646-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:08.977824 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:08.977605 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1852843-24f8-4260-acb4-e4c38514a646-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:09.135467 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:09.135434 5086 generic.go:356] "Generic (PLEG): container finished" podID="d1852843-24f8-4260-acb4-e4c38514a646" containerID="a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0" exitCode=0 Nov 03 11:28:09.135617 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:09.135490 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" Nov 03 11:28:09.135617 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:09.135490 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" event={"ID":"d1852843-24f8-4260-acb4-e4c38514a646","Type":"ContainerDied","Data":"a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0"} Nov 03 11:28:09.135617 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:09.135555 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b" event={"ID":"d1852843-24f8-4260-acb4-e4c38514a646","Type":"ContainerDied","Data":"e4f2cc302de218c018395c14f3ae34414428bcf05f46704a7c2f3f4c83300f64"} Nov 03 11:28:09.135617 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:09.135585 5086 scope.go:122] "RemoveContainer" containerID="a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0" Nov 03 11:28:09.146265 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:09.146245 5086 scope.go:122] "RemoveContainer" containerID="a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0" Nov 03 11:28:09.146544 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:28:09.146528 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0\": container with ID starting with a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0 not found: ID does not exist" containerID="a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0" Nov 03 11:28:09.146583 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:09.146551 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0"} err="failed to get container status \"a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0\": rpc error: code = NotFound desc = could not find container \"a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0\": container with ID starting with a333103e4c43f559d305e279ad7a4eb76ad18546f602e567ff57fa05750c08e0 not found: ID does not exist" Nov 03 11:28:09.160259 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:09.160231 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b"] Nov 03 11:28:09.164019 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:09.163999 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cbeebd-859c98bfb9-lzd8b"] Nov 03 11:28:10.172164 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:10.172132 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1852843-24f8-4260-acb4-e4c38514a646" path="/var/lib/kubelet/pods/d1852843-24f8-4260-acb4-e4c38514a646/volumes" Nov 03 11:28:21.818683 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:21.818647 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr"] Nov 03 11:28:21.819184 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:21.818903 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" podUID="cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" containerName="maas-api" containerID="cri-o://7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b" gracePeriod=30 Nov 03 11:28:21.931034 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:21.930995 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" podUID="cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.165:8443/health\": dial tcp 10.132.0.165:8443: connect: connection refused" Nov 03 11:28:22.137146 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.136786 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 11:28:22.239951 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.239913 5086 generic.go:356] "Generic (PLEG): container finished" podID="cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" containerID="7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b" exitCode=0 Nov 03 11:28:22.239951 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.239951 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" Nov 03 11:28:22.240243 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.239995 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" event={"ID":"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6","Type":"ContainerDied","Data":"7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b"} Nov 03 11:28:22.240243 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.240056 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr" event={"ID":"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6","Type":"ContainerDied","Data":"a280a2978a7456d8748486e5819728ad3d102f9d9a780bd261e51c10bcdac510"} Nov 03 11:28:22.240243 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.240083 5086 scope.go:122] "RemoveContainer" containerID="7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b" Nov 03 11:28:22.250756 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.250738 5086 scope.go:122] "RemoveContainer" containerID="7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b" Nov 03 11:28:22.251036 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:28:22.251019 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b\": container with ID starting with 7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b not found: ID does not exist" containerID="7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b" Nov 03 11:28:22.251077 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.251045 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b"} err="failed to get container status \"7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b\": rpc error: code = NotFound desc = could not find container \"7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b\": container with ID starting with 7d6851982eb994336e9b20dfaba988a7a19be80db06cad0a4de6e81461bae76b not found: ID does not exist" Nov 03 11:28:22.286578 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.286552 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-metrics-certs\") pod \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " Nov 03 11:28:22.286743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.286629 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-openshift-service-ca\") pod \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " Nov 03 11:28:22.286743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.286683 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-maas-api-tls\") pod \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " Nov 03 11:28:22.286743 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.286729 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-kube-api-access-qwrkd\" (UniqueName: \"kubernetes.io/projected/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-kube-api-access-qwrkd\") pod \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\" (UID: \"cd71a4f4-6950-42dc-8f1f-502cdf5c64e6\") " Nov 03 11:28:22.287060 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.287028 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-openshift-service-ca" pod "cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" (UID: "cd71a4f4-6950-42dc-8f1f-502cdf5c64e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:28:22.287188 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.287138 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:22.288837 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.288813 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-maas-api-tls" pod "cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" (UID: "cd71a4f4-6950-42dc-8f1f-502cdf5c64e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:28:22.288973 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.288914 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-metrics-certs" pod "cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" (UID: "cd71a4f4-6950-42dc-8f1f-502cdf5c64e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:28:22.289032 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.289001 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-kube-api-access-qwrkd" pod "cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" (UID: "cd71a4f4-6950-42dc-8f1f-502cdf5c64e6"). InnerVolumeSpecName "kube-api-access-qwrkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:28:22.389110 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.389020 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:22.389110 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.389055 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrkd\" (UniqueName: \"kubernetes.io/projected/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-kube-api-access-qwrkd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:22.389110 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.389066 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:22.575395 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.575363 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr"] Nov 03 11:28:22.576399 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:22.576375 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e11de4-79867bdf89-7qbrr"] Nov 03 11:28:24.186280 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:24.185481 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" path="/var/lib/kubelet/pods/cd71a4f4-6950-42dc-8f1f-502cdf5c64e6/volumes" Nov 03 11:28:24.281114 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:28:24.281076 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:28:41.681345 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:41.677479 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7"] Nov 03 11:28:41.681345 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:41.677933 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" podUID="ee98b2af-6d15-403a-99b6-02aeb95b98af" containerName="maas-api" containerID="cri-o://b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4" gracePeriod=30 Nov 03 11:28:42.016564 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.016529 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:28:42.087844 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.087792 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-maas-api-tls\") pod \"ee98b2af-6d15-403a-99b6-02aeb95b98af\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " Nov 03 11:28:42.088123 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.087939 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee98b2af-6d15-403a-99b6-02aeb95b98af-kube-api-access-mzxhm\" (UniqueName: \"kubernetes.io/projected/ee98b2af-6d15-403a-99b6-02aeb95b98af-kube-api-access-mzxhm\") pod \"ee98b2af-6d15-403a-99b6-02aeb95b98af\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " Nov 03 11:28:42.088123 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.088022 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-metrics-certs\") pod \"ee98b2af-6d15-403a-99b6-02aeb95b98af\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " Nov 03 11:28:42.088270 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.088181 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee98b2af-6d15-403a-99b6-02aeb95b98af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee98b2af-6d15-403a-99b6-02aeb95b98af-openshift-service-ca\") pod \"ee98b2af-6d15-403a-99b6-02aeb95b98af\" (UID: \"ee98b2af-6d15-403a-99b6-02aeb95b98af\") " Nov 03 11:28:42.089162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.089130 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee98b2af-6d15-403a-99b6-02aeb95b98af-openshift-service-ca" pod "ee98b2af-6d15-403a-99b6-02aeb95b98af" (UID: "ee98b2af-6d15-403a-99b6-02aeb95b98af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:28:42.091918 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.091891 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-maas-api-tls" pod "ee98b2af-6d15-403a-99b6-02aeb95b98af" (UID: "ee98b2af-6d15-403a-99b6-02aeb95b98af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:28:42.092003 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.091919 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-metrics-certs" pod "ee98b2af-6d15-403a-99b6-02aeb95b98af" (UID: "ee98b2af-6d15-403a-99b6-02aeb95b98af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:28:42.092456 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.092434 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee98b2af-6d15-403a-99b6-02aeb95b98af-kube-api-access-mzxhm" pod "ee98b2af-6d15-403a-99b6-02aeb95b98af" (UID: "ee98b2af-6d15-403a-99b6-02aeb95b98af"). InnerVolumeSpecName "kube-api-access-mzxhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:28:42.209932 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.209874 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:42.209932 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.209913 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee98b2af-6d15-403a-99b6-02aeb95b98af-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:42.209932 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.209928 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee98b2af-6d15-403a-99b6-02aeb95b98af-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:42.210266 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.209945 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzxhm\" (UniqueName: \"kubernetes.io/projected/ee98b2af-6d15-403a-99b6-02aeb95b98af-kube-api-access-mzxhm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:42.215416 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.215384 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m"] Nov 03 11:28:42.390944 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.390909 5086 generic.go:356] "Generic (PLEG): container finished" podID="ee98b2af-6d15-403a-99b6-02aeb95b98af" containerID="b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4" exitCode=0 Nov 03 11:28:42.391120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.390974 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" Nov 03 11:28:42.391120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.390980 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" event={"ID":"ee98b2af-6d15-403a-99b6-02aeb95b98af","Type":"ContainerDied","Data":"b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4"} Nov 03 11:28:42.391120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.391099 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7" event={"ID":"ee98b2af-6d15-403a-99b6-02aeb95b98af","Type":"ContainerDied","Data":"caf8bcdb582bf7f761867ac364a83230991a84047ed5c43220c9c4b87843cb4e"} Nov 03 11:28:42.391120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.391120 5086 scope.go:122] "RemoveContainer" containerID="b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4" Nov 03 11:28:42.405164 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.405144 5086 scope.go:122] "RemoveContainer" containerID="b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4" Nov 03 11:28:42.405513 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:28:42.405480 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4\": container with ID starting with b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4 not found: ID does not exist" containerID="b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4" Nov 03 11:28:42.405614 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.405517 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4"} err="failed to get container status \"b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4\": rpc error: code = NotFound desc = could not find container \"b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4\": container with ID starting with b81b8247a7ca375b55d90dc3cdbb6e933ea20f3f9111f1c4aca1845eae4c5fc4 not found: ID does not exist" Nov 03 11:28:42.415685 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.415544 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7"] Nov 03 11:28:42.423027 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:42.422997 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b76c62-8696bdcfbc-79gr7"] Nov 03 11:28:43.225639 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:43.225609 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:28:43.232883 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:43.232860 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:28:43.253387 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:43.253367 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:28:43.258667 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:43.258649 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:28:44.175182 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:44.175145 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee98b2af-6d15-403a-99b6-02aeb95b98af" path="/var/lib/kubelet/pods/ee98b2af-6d15-403a-99b6-02aeb95b98af/volumes" Nov 03 11:28:48.282105 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:48.282065 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k"] Nov 03 11:28:50.242054 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:50.242016 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms"] Nov 03 11:28:50.263425 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:50.263390 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb"] Nov 03 11:28:52.920165 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.920121 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v"] Nov 03 11:28:52.921273 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921243 5086 cpu_manager.go:414] "containerMap: removing container" podUID="14889afa-093e-40af-93d5-28a889c72d6c" containerName="maas-api" Nov 03 11:28:52.921273 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921273 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14889afa-093e-40af-93d5-28a889c72d6c" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921310 5086 cpu_manager.go:414] "containerMap: removing container" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921319 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921334 5086 cpu_manager.go:414] "containerMap: removing container" podUID="a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921340 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921355 5086 cpu_manager.go:414] "containerMap: removing container" podUID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921362 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921376 5086 cpu_manager.go:414] "containerMap: removing container" podUID="eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921383 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921394 5086 cpu_manager.go:414] "containerMap: removing container" podUID="cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921402 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921422 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ee98b2af-6d15-403a-99b6-02aeb95b98af" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921429 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee98b2af-6d15-403a-99b6-02aeb95b98af" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921451 5086 cpu_manager.go:414] "containerMap: removing container" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921460 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921478 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d1852843-24f8-4260-acb4-e4c38514a646" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921485 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1852843-24f8-4260-acb4-e4c38514a646" containerName="maas-api" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921504 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerName="istio-proxy" Nov 03 11:28:52.921542 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921511 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerName="istio-proxy" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921699 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerName="maas-api" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921714 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d745bb74-c06c-47b9-8b0e-388da32dd45b" containerName="istio-proxy" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921734 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="14889afa-093e-40af-93d5-28a889c72d6c" containerName="maas-api" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921743 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="faeb4aef-28a7-447c-a5f9-7992a41382f9" containerName="istio-proxy" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921754 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c1b8dd-1f29-47a3-8a2b-a2f56526131c" containerName="maas-api" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921764 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerName="maas-api" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921773 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb7b2a6-ee19-4f89-8fca-23ba1c8fb3c2" containerName="maas-api" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921780 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd71a4f4-6950-42dc-8f1f-502cdf5c64e6" containerName="maas-api" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921791 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee98b2af-6d15-403a-99b6-02aeb95b98af" containerName="maas-api" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921802 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa80780-52f3-4910-b141-4fc80639cc6e" containerName="istio-proxy" Nov 03 11:28:52.922558 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.921811 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1852843-24f8-4260-acb4-e4c38514a646" containerName="maas-api" Nov 03 11:28:52.926341 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.926279 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:52.930816 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.930790 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v"] Nov 03 11:28:52.935011 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:52.934988 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce5555\"/\"sub-test-model-ce5555-kserve-self-signed-certs\"" Nov 03 11:28:53.067218 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.067181 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wd5\" (UniqueName: \"kubernetes.io/projected/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-kube-api-access-t4wd5\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.067449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.067250 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-model-cache\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.067517 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.067443 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-dshm\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.067582 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.067569 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tls-certs\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.067638 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.067616 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tmp-dir\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.067689 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.067672 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-home\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.169055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.169016 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-home\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.169254 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.169083 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wd5\" (UniqueName: \"kubernetes.io/projected/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-kube-api-access-t4wd5\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.169254 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.169139 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-model-cache\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.169254 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.169229 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-dshm\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.169504 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.169397 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tls-certs\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.169504 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.169446 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tmp-dir\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.169791 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.169760 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-model-cache\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.169930 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.169830 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-home\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.170009 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.169958 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tmp-dir\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.172053 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.171975 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-dshm\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.177720 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.177690 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tls-certs\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.177904 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.177881 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wd5\" (UniqueName: \"kubernetes.io/projected/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-kube-api-access-t4wd5\") pod \"sub-test-model-ce5555-kserve-77696994fc-wxd6v\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.243069 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.243037 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:53.530629 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:53.530605 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v"] Nov 03 11:28:53.531466 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:28:53.531440 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40abd8ce_e351_4f5d_8bad_b774e6bd47c3.slice/crio-5001aaf5f431c3eb3c203794d5fbbeb57485a05d110b101b5293f2da7ab37736 WatchSource:0}: Error finding container 5001aaf5f431c3eb3c203794d5fbbeb57485a05d110b101b5293f2da7ab37736: Status 404 returned error can't find the container with id 5001aaf5f431c3eb3c203794d5fbbeb57485a05d110b101b5293f2da7ab37736 Nov 03 11:28:54.276818 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:28:54.276777 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:28:54.510266 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:54.510227 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" event={"ID":"40abd8ce-e351-4f5d-8bad-b774e6bd47c3","Type":"ContainerStarted","Data":"2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d"} Nov 03 11:28:54.510477 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:54.510285 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" event={"ID":"40abd8ce-e351-4f5d-8bad-b774e6bd47c3","Type":"ContainerStarted","Data":"5001aaf5f431c3eb3c203794d5fbbeb57485a05d110b101b5293f2da7ab37736"} Nov 03 11:28:54.527449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:54.527352 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" podStartSLOduration=2.527336832 podStartE2EDuration="2.527336832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:28:54.525698404 +0000 UTC m=+3612.510569711" watchObservedRunningTime="2026-11-03 11:28:54.527336832 +0000 UTC m=+3612.512208114" Nov 03 11:28:55.242715 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:55.242672 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" podUID="692a59fd-e1ee-402e-91f0-ff267e768e24" containerName="payload-processing" containerID="cri-o://1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513" gracePeriod=25 Nov 03 11:28:55.244167 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:55.244141 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:28:55.264363 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:55.264313 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" podUID="0f294516-cbba-4b13-87f4-e8a9815f82c8" containerName="payload-pre-processing" containerID="cri-o://d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204" gracePeriod=25 Nov 03 11:28:57.251015 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.250948 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" podUID="b4ea433f-d697-460c-8e67-32b03caa1440" containerName="main" containerID="cri-o://511ce26580d4ea6ce4344066c78979225b8db97a8fcd119d88f3c19513b01b65" gracePeriod=45 Nov 03 11:28:57.532765 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.532735 5086 generic.go:356] "Generic (PLEG): container finished" podID="b4ea433f-d697-460c-8e67-32b03caa1440" containerID="511ce26580d4ea6ce4344066c78979225b8db97a8fcd119d88f3c19513b01b65" exitCode=0 Nov 03 11:28:57.532935 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.532815 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" event={"ID":"b4ea433f-d697-460c-8e67-32b03caa1440","Type":"ContainerDied","Data":"511ce26580d4ea6ce4344066c78979225b8db97a8fcd119d88f3c19513b01b65"} Nov 03 11:28:57.532935 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.532860 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" event={"ID":"b4ea433f-d697-460c-8e67-32b03caa1440","Type":"ContainerDied","Data":"bf5794035e1b14d53c8df8b90b475fbe231a7830ccb65abc0f59ee4383fbf896"} Nov 03 11:28:57.532935 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.532884 5086 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf5794035e1b14d53c8df8b90b475fbe231a7830ccb65abc0f59ee4383fbf896" Nov 03 11:28:57.546464 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.546437 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:28:57.644854 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.644814 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ea433f-d697-460c-8e67-32b03caa1440-kube-api-access-hfdtx\" (UniqueName: \"kubernetes.io/projected/b4ea433f-d697-460c-8e67-32b03caa1440-kube-api-access-hfdtx\") pod \"b4ea433f-d697-460c-8e67-32b03caa1440\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " Nov 03 11:28:57.645045 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.644973 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-tmp-dir\") pod \"b4ea433f-d697-460c-8e67-32b03caa1440\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " Nov 03 11:28:57.645207 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.645183 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-dshm\") pod \"b4ea433f-d697-460c-8e67-32b03caa1440\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " Nov 03 11:28:57.645385 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.645266 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-tmp-dir" pod "b4ea433f-d697-460c-8e67-32b03caa1440" (UID: "b4ea433f-d697-460c-8e67-32b03caa1440"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:28:57.645483 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.645454 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ea433f-d697-460c-8e67-32b03caa1440-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea433f-d697-460c-8e67-32b03caa1440-tls-certs\") pod \"b4ea433f-d697-460c-8e67-32b03caa1440\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " Nov 03 11:28:57.645581 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.645564 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-model-cache\") pod \"b4ea433f-d697-460c-8e67-32b03caa1440\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " Nov 03 11:28:57.645650 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.645611 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-home\") pod \"b4ea433f-d697-460c-8e67-32b03caa1440\" (UID: \"b4ea433f-d697-460c-8e67-32b03caa1440\") " Nov 03 11:28:57.645913 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.645854 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-model-cache" pod "b4ea433f-d697-460c-8e67-32b03caa1440" (UID: "b4ea433f-d697-460c-8e67-32b03caa1440"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:28:57.646013 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.645958 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-home" pod "b4ea433f-d697-460c-8e67-32b03caa1440" (UID: "b4ea433f-d697-460c-8e67-32b03caa1440"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:28:57.646268 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.646246 5086 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:57.646390 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.646273 5086 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:57.646390 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.646290 5086 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:57.647450 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.647426 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-dshm" pod "b4ea433f-d697-460c-8e67-32b03caa1440" (UID: "b4ea433f-d697-460c-8e67-32b03caa1440"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:28:57.647601 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.647582 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ea433f-d697-460c-8e67-32b03caa1440-kube-api-access-hfdtx" pod "b4ea433f-d697-460c-8e67-32b03caa1440" (UID: "b4ea433f-d697-460c-8e67-32b03caa1440"). InnerVolumeSpecName "kube-api-access-hfdtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:28:57.647690 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.647672 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ea433f-d697-460c-8e67-32b03caa1440-tls-certs" pod "b4ea433f-d697-460c-8e67-32b03caa1440" (UID: "b4ea433f-d697-460c-8e67-32b03caa1440"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:28:57.747426 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.747388 5086 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea433f-d697-460c-8e67-32b03caa1440-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:57.747426 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.747428 5086 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea433f-d697-460c-8e67-32b03caa1440-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:57.747633 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:57.747442 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfdtx\" (UniqueName: \"kubernetes.io/projected/b4ea433f-d697-460c-8e67-32b03caa1440-kube-api-access-hfdtx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:28:58.541029 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:58.540950 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m" Nov 03 11:28:58.569962 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:58.569927 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m"] Nov 03 11:28:58.575510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:28:58.575483 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/rate-test-model-ce5555-kserve-5f6f467f7-7s96m"] Nov 03 11:29:00.172109 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:00.172057 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ea433f-d697-460c-8e67-32b03caa1440" path="/var/lib/kubelet/pods/b4ea433f-d697-460c-8e67-32b03caa1440/volumes" Nov 03 11:29:01.486862 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:01.486821 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" podUID="0f294516-cbba-4b13-87f4-e8a9815f82c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 11:29:01.486862 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:01.486850 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" podUID="692a59fd-e1ee-402e-91f0-ff267e768e24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 11:29:03.243949 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.243915 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:29:03.258082 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.258047 5086 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:29:03.307513 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.307467 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" podUID="390485cb-9990-4cb9-8a8e-38e57f80b2cf" containerName="main" containerID="cri-o://8708997c5f6fac6e1edc2cf42d51f67a99a0065eac5a3fe5d5ebc54e87281781" gracePeriod=45 Nov 03 11:29:03.582619 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.582586 5086 generic.go:356] "Generic (PLEG): container finished" podID="390485cb-9990-4cb9-8a8e-38e57f80b2cf" containerID="8708997c5f6fac6e1edc2cf42d51f67a99a0065eac5a3fe5d5ebc54e87281781" exitCode=0 Nov 03 11:29:03.582825 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.582667 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" event={"ID":"390485cb-9990-4cb9-8a8e-38e57f80b2cf","Type":"ContainerDied","Data":"8708997c5f6fac6e1edc2cf42d51f67a99a0065eac5a3fe5d5ebc54e87281781"} Nov 03 11:29:03.594475 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.594448 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:29:03.631782 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.631752 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:29:03.644465 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.644435 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-model-cache\") pod \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " Nov 03 11:29:03.644663 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.644502 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tmp-dir\") pod \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " Nov 03 11:29:03.644663 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.644532 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tls-certs\") pod \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " Nov 03 11:29:03.644663 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.644603 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390485cb-9990-4cb9-8a8e-38e57f80b2cf-kube-api-access-xrj7t\" (UniqueName: \"kubernetes.io/projected/390485cb-9990-4cb9-8a8e-38e57f80b2cf-kube-api-access-xrj7t\") pod \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " Nov 03 11:29:03.644663 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.644624 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-dshm\") pod \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " Nov 03 11:29:03.644908 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.644742 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-home\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-home\") pod \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\" (UID: \"390485cb-9990-4cb9-8a8e-38e57f80b2cf\") " Nov 03 11:29:03.645786 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.645740 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-model-cache" pod "390485cb-9990-4cb9-8a8e-38e57f80b2cf" (UID: "390485cb-9990-4cb9-8a8e-38e57f80b2cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:29:03.645917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.645896 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-home" pod "390485cb-9990-4cb9-8a8e-38e57f80b2cf" (UID: "390485cb-9990-4cb9-8a8e-38e57f80b2cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:29:03.646044 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.646020 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tmp-dir" pod "390485cb-9990-4cb9-8a8e-38e57f80b2cf" (UID: "390485cb-9990-4cb9-8a8e-38e57f80b2cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:29:03.647873 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.647832 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-dshm" pod "390485cb-9990-4cb9-8a8e-38e57f80b2cf" (UID: "390485cb-9990-4cb9-8a8e-38e57f80b2cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:29:03.650338 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.649782 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390485cb-9990-4cb9-8a8e-38e57f80b2cf-kube-api-access-xrj7t" pod "390485cb-9990-4cb9-8a8e-38e57f80b2cf" (UID: "390485cb-9990-4cb9-8a8e-38e57f80b2cf"). InnerVolumeSpecName "kube-api-access-xrj7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:29:03.655179 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.655153 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tls-certs" pod "390485cb-9990-4cb9-8a8e-38e57f80b2cf" (UID: "390485cb-9990-4cb9-8a8e-38e57f80b2cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:29:03.751491 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.751452 5086 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:03.751491 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.751491 5086 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:03.751964 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.751507 5086 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390485cb-9990-4cb9-8a8e-38e57f80b2cf-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:03.751964 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.751521 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrj7t\" (UniqueName: \"kubernetes.io/projected/390485cb-9990-4cb9-8a8e-38e57f80b2cf-kube-api-access-xrj7t\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:03.751964 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.751536 5086 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:03.751964 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:03.751549 5086 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390485cb-9990-4cb9-8a8e-38e57f80b2cf-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:04.592744 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:04.592711 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" Nov 03 11:29:04.593330 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:04.592744 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k" event={"ID":"390485cb-9990-4cb9-8a8e-38e57f80b2cf","Type":"ContainerDied","Data":"127df9c39658aaff90d7ee475e58dee94ded310faf8acf100e9a9014e117312e"} Nov 03 11:29:04.593330 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:04.592793 5086 scope.go:122] "RemoveContainer" containerID="8708997c5f6fac6e1edc2cf42d51f67a99a0065eac5a3fe5d5ebc54e87281781" Nov 03 11:29:04.611700 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:04.611673 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k"] Nov 03 11:29:04.618634 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:04.618609 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/rate-test-model-836589-kserve-7d6f5579cc-dbp9k"] Nov 03 11:29:06.175492 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:06.175459 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390485cb-9990-4cb9-8a8e-38e57f80b2cf" path="/var/lib/kubelet/pods/390485cb-9990-4cb9-8a8e-38e57f80b2cf/volumes" Nov 03 11:29:10.111017 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.110977 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c"] Nov 03 11:29:10.111587 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.111220 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" podUID="9fcdd3bd-363b-4747-81c0-8bad0aee4a72" containerName="maas-api" containerID="cri-o://39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e" gracePeriod=30 Nov 03 11:29:10.467758 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.467715 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 11:29:10.547572 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.547536 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-openshift-service-ca\") pod \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " Nov 03 11:29:10.547765 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.547624 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-metrics-certs\") pod \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " Nov 03 11:29:10.547765 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.547718 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls\") pod \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " Nov 03 11:29:10.547883 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.547848 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-kube-api-access-s4blg\" (UniqueName: \"kubernetes.io/projected/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-kube-api-access-s4blg\") pod \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\" (UID: \"9fcdd3bd-363b-4747-81c0-8bad0aee4a72\") " Nov 03 11:29:10.547972 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.547943 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-openshift-service-ca" pod "9fcdd3bd-363b-4747-81c0-8bad0aee4a72" (UID: "9fcdd3bd-363b-4747-81c0-8bad0aee4a72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:29:10.548363 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.548343 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:10.550007 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.549974 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-kube-api-access-s4blg" pod "9fcdd3bd-363b-4747-81c0-8bad0aee4a72" (UID: "9fcdd3bd-363b-4747-81c0-8bad0aee4a72"). InnerVolumeSpecName "kube-api-access-s4blg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:29:10.550338 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.550280 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-metrics-certs" pod "9fcdd3bd-363b-4747-81c0-8bad0aee4a72" (UID: "9fcdd3bd-363b-4747-81c0-8bad0aee4a72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:29:10.550421 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.550312 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls" pod "9fcdd3bd-363b-4747-81c0-8bad0aee4a72" (UID: "9fcdd3bd-363b-4747-81c0-8bad0aee4a72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:29:10.644518 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.644477 5086 generic.go:356] "Generic (PLEG): container finished" podID="9fcdd3bd-363b-4747-81c0-8bad0aee4a72" containerID="39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e" exitCode=0 Nov 03 11:29:10.644710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.644529 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" Nov 03 11:29:10.644710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.644555 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" event={"ID":"9fcdd3bd-363b-4747-81c0-8bad0aee4a72","Type":"ContainerDied","Data":"39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e"} Nov 03 11:29:10.644710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.644609 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c" event={"ID":"9fcdd3bd-363b-4747-81c0-8bad0aee4a72","Type":"ContainerDied","Data":"f474f60eafdc853d033cb9fe43cbf036d399ee1ad3badfff3a209e70e5e9e94d"} Nov 03 11:29:10.644710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.644631 5086 scope.go:122] "RemoveContainer" containerID="39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e" Nov 03 11:29:10.649785 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.649750 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:10.649929 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.649791 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4blg\" (UniqueName: \"kubernetes.io/projected/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-kube-api-access-s4blg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:10.649929 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.649806 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcdd3bd-363b-4747-81c0-8bad0aee4a72-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:10.657790 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.657755 5086 scope.go:122] "RemoveContainer" containerID="39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e" Nov 03 11:29:10.658131 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:29:10.658103 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e\": container with ID starting with 39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e not found: ID does not exist" containerID="39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e" Nov 03 11:29:10.658234 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.658130 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e"} err="failed to get container status \"39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e\": rpc error: code = NotFound desc = could not find container \"39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e\": container with ID starting with 39560f4f54d9602bc59f43be33f677022a8ea7a409c52e28c2c1f619ba7fca9e not found: ID does not exist" Nov 03 11:29:10.702488 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.702402 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c"] Nov 03 11:29:10.706024 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:10.705999 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e92a08-76b94f5c5f-fgh6c"] Nov 03 11:29:11.486810 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:11.486768 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" podUID="0f294516-cbba-4b13-87f4-e8a9815f82c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 11:29:11.487241 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:11.486836 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" podUID="692a59fd-e1ee-402e-91f0-ff267e768e24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 11:29:12.181281 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:12.181244 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcdd3bd-363b-4747-81c0-8bad0aee4a72" path="/var/lib/kubelet/pods/9fcdd3bd-363b-4747-81c0-8bad0aee4a72/volumes" Nov 03 11:29:15.543029 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.543001 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 11:29:15.609162 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.609122 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 11:29:15.614946 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.614914 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692a59fd-e1ee-402e-91f0-ff267e768e24-kube-api-access-9qbt9\" (UniqueName: \"kubernetes.io/projected/692a59fd-e1ee-402e-91f0-ff267e768e24-kube-api-access-9qbt9\") pod \"692a59fd-e1ee-402e-91f0-ff267e768e24\" (UID: \"692a59fd-e1ee-402e-91f0-ff267e768e24\") " Nov 03 11:29:15.615106 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.615029 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692a59fd-e1ee-402e-91f0-ff267e768e24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a59fd-e1ee-402e-91f0-ff267e768e24-plugins-config-volume\") pod \"692a59fd-e1ee-402e-91f0-ff267e768e24\" (UID: \"692a59fd-e1ee-402e-91f0-ff267e768e24\") " Nov 03 11:29:15.615502 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.615464 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692a59fd-e1ee-402e-91f0-ff267e768e24-plugins-config-volume" pod "692a59fd-e1ee-402e-91f0-ff267e768e24" (UID: "692a59fd-e1ee-402e-91f0-ff267e768e24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:29:15.615688 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.615663 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692a59fd-e1ee-402e-91f0-ff267e768e24-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:15.617181 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.617157 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692a59fd-e1ee-402e-91f0-ff267e768e24-kube-api-access-9qbt9" pod "692a59fd-e1ee-402e-91f0-ff267e768e24" (UID: "692a59fd-e1ee-402e-91f0-ff267e768e24"). InnerVolumeSpecName "kube-api-access-9qbt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:29:15.681097 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.681064 5086 generic.go:356] "Generic (PLEG): container finished" podID="692a59fd-e1ee-402e-91f0-ff267e768e24" containerID="1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513" exitCode=0 Nov 03 11:29:15.681306 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.681107 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" Nov 03 11:29:15.681306 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.681147 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" event={"ID":"692a59fd-e1ee-402e-91f0-ff267e768e24","Type":"ContainerDied","Data":"1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513"} Nov 03 11:29:15.681306 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.681195 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms" event={"ID":"692a59fd-e1ee-402e-91f0-ff267e768e24","Type":"ContainerDied","Data":"8b405c8494a3a678ad16ecb4bd4689c4f80a459f1cbf6377e96d9c968840f590"} Nov 03 11:29:15.681306 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.681219 5086 scope.go:122] "RemoveContainer" containerID="1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513" Nov 03 11:29:15.682819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.682796 5086 generic.go:356] "Generic (PLEG): container finished" podID="0f294516-cbba-4b13-87f4-e8a9815f82c8" containerID="d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204" exitCode=0 Nov 03 11:29:15.682944 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.682855 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" Nov 03 11:29:15.682944 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.682886 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" event={"ID":"0f294516-cbba-4b13-87f4-e8a9815f82c8","Type":"ContainerDied","Data":"d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204"} Nov 03 11:29:15.682944 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.682939 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb" event={"ID":"0f294516-cbba-4b13-87f4-e8a9815f82c8","Type":"ContainerDied","Data":"4a2236bcab6ecf71949f17e6e344f184d5d292c001f7063efc5dddb5dc0b0e1e"} Nov 03 11:29:15.691681 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.691665 5086 scope.go:122] "RemoveContainer" containerID="1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513" Nov 03 11:29:15.691947 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:29:15.691921 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513\": container with ID starting with 1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513 not found: ID does not exist" containerID="1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513" Nov 03 11:29:15.692042 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.691952 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513"} err="failed to get container status \"1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513\": rpc error: code = NotFound desc = could not find container \"1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513\": container with ID starting with 1ce57800116d47d5972afbe968b4080f86c6ea9546c803cd674497df881ce513 not found: ID does not exist" Nov 03 11:29:15.692042 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.691966 5086 scope.go:122] "RemoveContainer" containerID="d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204" Nov 03 11:29:15.701547 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.701525 5086 scope.go:122] "RemoveContainer" containerID="d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204" Nov 03 11:29:15.701991 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:29:15.701968 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204\": container with ID starting with d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204 not found: ID does not exist" containerID="d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204" Nov 03 11:29:15.702083 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.701995 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204"} err="failed to get container status \"d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204\": rpc error: code = NotFound desc = could not find container \"d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204\": container with ID starting with d12b2b1f7c5807fe7613bfde77ec90153446ae7f3c91df1e7c2b345d2405d204 not found: ID does not exist" Nov 03 11:29:15.705211 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.705191 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms"] Nov 03 11:29:15.708016 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.707997 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e78c33-75cd6ff47c-wbvms"] Nov 03 11:29:15.717038 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.717015 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f294516-cbba-4b13-87f4-e8a9815f82c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f294516-cbba-4b13-87f4-e8a9815f82c8-plugins-config-volume\") pod \"0f294516-cbba-4b13-87f4-e8a9815f82c8\" (UID: \"0f294516-cbba-4b13-87f4-e8a9815f82c8\") " Nov 03 11:29:15.717150 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.717080 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f294516-cbba-4b13-87f4-e8a9815f82c8-kube-api-access-gcbn6\" (UniqueName: \"kubernetes.io/projected/0f294516-cbba-4b13-87f4-e8a9815f82c8-kube-api-access-gcbn6\") pod \"0f294516-cbba-4b13-87f4-e8a9815f82c8\" (UID: \"0f294516-cbba-4b13-87f4-e8a9815f82c8\") " Nov 03 11:29:15.717387 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.717367 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f294516-cbba-4b13-87f4-e8a9815f82c8-plugins-config-volume" pod "0f294516-cbba-4b13-87f4-e8a9815f82c8" (UID: "0f294516-cbba-4b13-87f4-e8a9815f82c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:29:15.717551 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.717534 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f294516-cbba-4b13-87f4-e8a9815f82c8-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:15.717609 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.717558 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qbt9\" (UniqueName: \"kubernetes.io/projected/692a59fd-e1ee-402e-91f0-ff267e768e24-kube-api-access-9qbt9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:15.719016 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.718997 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f294516-cbba-4b13-87f4-e8a9815f82c8-kube-api-access-gcbn6" pod "0f294516-cbba-4b13-87f4-e8a9815f82c8" (UID: "0f294516-cbba-4b13-87f4-e8a9815f82c8"). InnerVolumeSpecName "kube-api-access-gcbn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:29:15.819071 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:15.819037 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcbn6\" (UniqueName: \"kubernetes.io/projected/0f294516-cbba-4b13-87f4-e8a9815f82c8-kube-api-access-gcbn6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:16.008970 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:16.008935 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb"] Nov 03 11:29:16.012633 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:16.012607 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e78c33-6497cb76ff-jhrwb"] Nov 03 11:29:16.171890 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:16.171808 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f294516-cbba-4b13-87f4-e8a9815f82c8" path="/var/lib/kubelet/pods/0f294516-cbba-4b13-87f4-e8a9815f82c8/volumes" Nov 03 11:29:16.172272 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:16.172254 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692a59fd-e1ee-402e-91f0-ff267e768e24" path="/var/lib/kubelet/pods/692a59fd-e1ee-402e-91f0-ff267e768e24/volumes" Nov 03 11:29:17.263821 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:17.263785 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9"] Nov 03 11:29:17.702377 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:17.702289 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" podUID="955750bc-952d-4b29-b6e8-a4c119428fba" containerName="maas-api" containerID="cri-o://6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e" gracePeriod=30 Nov 03 11:29:18.019706 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.019674 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 11:29:18.064247 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.064204 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-maas-api-tls\") pod \"955750bc-952d-4b29-b6e8-a4c119428fba\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " Nov 03 11:29:18.064418 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.064394 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-metrics-certs\") pod \"955750bc-952d-4b29-b6e8-a4c119428fba\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " Nov 03 11:29:18.064517 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.064497 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955750bc-952d-4b29-b6e8-a4c119428fba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955750bc-952d-4b29-b6e8-a4c119428fba-openshift-service-ca\") pod \"955750bc-952d-4b29-b6e8-a4c119428fba\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " Nov 03 11:29:18.064651 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.064632 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955750bc-952d-4b29-b6e8-a4c119428fba-kube-api-access-prmmj\" (UniqueName: \"kubernetes.io/projected/955750bc-952d-4b29-b6e8-a4c119428fba-kube-api-access-prmmj\") pod \"955750bc-952d-4b29-b6e8-a4c119428fba\" (UID: \"955750bc-952d-4b29-b6e8-a4c119428fba\") " Nov 03 11:29:18.067159 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.066032 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955750bc-952d-4b29-b6e8-a4c119428fba-openshift-service-ca" pod "955750bc-952d-4b29-b6e8-a4c119428fba" (UID: "955750bc-952d-4b29-b6e8-a4c119428fba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:29:18.073829 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.068845 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-maas-api-tls" pod "955750bc-952d-4b29-b6e8-a4c119428fba" (UID: "955750bc-952d-4b29-b6e8-a4c119428fba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:29:18.073829 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.073034 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-metrics-certs" pod "955750bc-952d-4b29-b6e8-a4c119428fba" (UID: "955750bc-952d-4b29-b6e8-a4c119428fba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:29:18.073829 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.073287 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955750bc-952d-4b29-b6e8-a4c119428fba-kube-api-access-prmmj" pod "955750bc-952d-4b29-b6e8-a4c119428fba" (UID: "955750bc-952d-4b29-b6e8-a4c119428fba"). InnerVolumeSpecName "kube-api-access-prmmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:29:18.166346 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.166288 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:18.166346 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.166348 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955750bc-952d-4b29-b6e8-a4c119428fba-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:18.167527 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.166363 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prmmj\" (UniqueName: \"kubernetes.io/projected/955750bc-952d-4b29-b6e8-a4c119428fba-kube-api-access-prmmj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:18.167527 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.166378 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955750bc-952d-4b29-b6e8-a4c119428fba-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:18.709527 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.709492 5086 generic.go:356] "Generic (PLEG): container finished" podID="955750bc-952d-4b29-b6e8-a4c119428fba" containerID="6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e" exitCode=0 Nov 03 11:29:18.710024 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.709538 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" Nov 03 11:29:18.710024 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.709575 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" event={"ID":"955750bc-952d-4b29-b6e8-a4c119428fba","Type":"ContainerDied","Data":"6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e"} Nov 03 11:29:18.710024 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.709638 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9" event={"ID":"955750bc-952d-4b29-b6e8-a4c119428fba","Type":"ContainerDied","Data":"cc3d7469b83f729160e11c7636453b30f7068119d19045919e65646562debd12"} Nov 03 11:29:18.710024 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.709669 5086 scope.go:122] "RemoveContainer" containerID="6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e" Nov 03 11:29:18.720493 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.720472 5086 scope.go:122] "RemoveContainer" containerID="6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e" Nov 03 11:29:18.720804 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:29:18.720779 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e\": container with ID starting with 6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e not found: ID does not exist" containerID="6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e" Nov 03 11:29:18.720883 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.720813 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e"} err="failed to get container status \"6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e\": rpc error: code = NotFound desc = could not find container \"6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e\": container with ID starting with 6c2a461a99892500a26c268d2acdbf55576e88e3763fc86ac9142ba42a27a47e not found: ID does not exist" Nov 03 11:29:18.736602 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.736469 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9"] Nov 03 11:29:18.740770 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:18.740743 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e78c33-64db578bd8-ppfs9"] Nov 03 11:29:20.171711 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:20.171678 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955750bc-952d-4b29-b6e8-a4c119428fba" path="/var/lib/kubelet/pods/955750bc-952d-4b29-b6e8-a4c119428fba/volumes" Nov 03 11:29:24.275693 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:29:24.275609 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:29:24.848196 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:24.848164 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw"] Nov 03 11:29:24.848399 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:24.848371 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" containerID="cri-o://9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a" gracePeriod=30 Nov 03 11:29:30.126907 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.126869 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 11:29:30.234191 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234110 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-token\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-token\") pod \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " Nov 03 11:29:30.234191 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234163 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-podinfo\") pod \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " Nov 03 11:29:30.234450 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234197 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-data\") pod \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " Nov 03 11:29:30.234450 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234234 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-socket\") pod \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " Nov 03 11:29:30.234450 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234307 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istiod-ca-cert\") pod \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " Nov 03 11:29:30.234450 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234381 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-credential-socket\") pod \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " Nov 03 11:29:30.234665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234566 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-envoy\") pod \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " Nov 03 11:29:30.234665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234585 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-socket" pod "f053e8e0-87a9-49fc-ae65-5c07b771b89e" (UID: "f053e8e0-87a9-49fc-ae65-5c07b771b89e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:29:30.234665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234598 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-data" pod "f053e8e0-87a9-49fc-ae65-5c07b771b89e" (UID: "f053e8e0-87a9-49fc-ae65-5c07b771b89e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:29:30.234665 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234598 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-kube-api-access-tpdz8\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-kube-api-access-tpdz8\") pod \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " Nov 03 11:29:30.234867 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234665 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-certs\") pod \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\" (UID: \"f053e8e0-87a9-49fc-ae65-5c07b771b89e\") " Nov 03 11:29:30.234867 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.234843 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-credential-socket" pod "f053e8e0-87a9-49fc-ae65-5c07b771b89e" (UID: "f053e8e0-87a9-49fc-ae65-5c07b771b89e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:29:30.235263 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.235220 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-certs" pod "f053e8e0-87a9-49fc-ae65-5c07b771b89e" (UID: "f053e8e0-87a9-49fc-ae65-5c07b771b89e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:29:30.235425 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.235361 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:30.235425 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.235381 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:30.235425 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.235396 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:30.235614 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.235506 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istiod-ca-cert" pod "f053e8e0-87a9-49fc-ae65-5c07b771b89e" (UID: "f053e8e0-87a9-49fc-ae65-5c07b771b89e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:29:30.237161 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.237022 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-token" pod "f053e8e0-87a9-49fc-ae65-5c07b771b89e" (UID: "f053e8e0-87a9-49fc-ae65-5c07b771b89e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:29:30.237582 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.237555 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-podinfo" pod "f053e8e0-87a9-49fc-ae65-5c07b771b89e" (UID: "f053e8e0-87a9-49fc-ae65-5c07b771b89e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:29:30.238113 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.238092 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-envoy" pod "f053e8e0-87a9-49fc-ae65-5c07b771b89e" (UID: "f053e8e0-87a9-49fc-ae65-5c07b771b89e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:29:30.238428 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.238408 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-kube-api-access-tpdz8" pod "f053e8e0-87a9-49fc-ae65-5c07b771b89e" (UID: "f053e8e0-87a9-49fc-ae65-5c07b771b89e"). InnerVolumeSpecName "kube-api-access-tpdz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:29:30.337150 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.337091 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:30.337150 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.337120 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:30.337150 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.337131 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpdz8\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-kube-api-access-tpdz8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:30.337150 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.337145 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f053e8e0-87a9-49fc-ae65-5c07b771b89e-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:30.337150 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.337155 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:30.337150 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.337164 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f053e8e0-87a9-49fc-ae65-5c07b771b89e-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:30.794627 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.794590 5086 generic.go:356] "Generic (PLEG): container finished" podID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerID="9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a" exitCode=0 Nov 03 11:29:30.794792 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.794638 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" event={"ID":"f053e8e0-87a9-49fc-ae65-5c07b771b89e","Type":"ContainerDied","Data":"9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a"} Nov 03 11:29:30.794792 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.794687 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" event={"ID":"f053e8e0-87a9-49fc-ae65-5c07b771b89e","Type":"ContainerDied","Data":"07eaeeeecc74925a903488912eb37c29a6f09bf68fca3cdc408ccf899b7770bc"} Nov 03 11:29:30.794792 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.794642 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw" Nov 03 11:29:30.794946 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.794717 5086 scope.go:122] "RemoveContainer" containerID="9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a" Nov 03 11:29:30.806858 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.806831 5086 scope.go:122] "RemoveContainer" containerID="9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a" Nov 03 11:29:30.807164 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:29:30.807142 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a\": container with ID starting with 9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a not found: ID does not exist" containerID="9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a" Nov 03 11:29:30.807249 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.807173 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a"} err="failed to get container status \"9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a\": rpc error: code = NotFound desc = could not find container \"9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a\": container with ID starting with 9bdbad06ad8f055ea2ec96e08eecca26684ace4b75fbdd84f839b642a9aa193a not found: ID does not exist" Nov 03 11:29:30.819016 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.818988 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw"] Nov 03 11:29:30.829410 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:30.829382 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-rvtbw"] Nov 03 11:29:32.173531 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:32.173489 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" path="/var/lib/kubelet/pods/f053e8e0-87a9-49fc-ae65-5c07b771b89e/volumes" Nov 03 11:29:35.786183 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:35.786147 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8"] Nov 03 11:29:35.786833 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:35.786468 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" podUID="c9670a7d-3aa9-4225-9cbb-783d6dcc4734" containerName="maas-api" containerID="cri-o://72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc" gracePeriod=30 Nov 03 11:29:36.104306 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.104266 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 11:29:36.267598 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.267567 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-metrics-certs\") pod \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " Nov 03 11:29:36.267792 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.267621 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-kube-api-access-mj88b\" (UniqueName: \"kubernetes.io/projected/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-kube-api-access-mj88b\") pod \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " Nov 03 11:29:36.267792 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.267684 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-openshift-service-ca\") pod \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " Nov 03 11:29:36.267792 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.267772 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-maas-api-tls\") pod \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\" (UID: \"c9670a7d-3aa9-4225-9cbb-783d6dcc4734\") " Nov 03 11:29:36.268110 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.268075 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-openshift-service-ca" pod "c9670a7d-3aa9-4225-9cbb-783d6dcc4734" (UID: "c9670a7d-3aa9-4225-9cbb-783d6dcc4734"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:29:36.268312 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.268279 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:36.269956 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.269931 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-metrics-certs" pod "c9670a7d-3aa9-4225-9cbb-783d6dcc4734" (UID: "c9670a7d-3aa9-4225-9cbb-783d6dcc4734"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:29:36.270058 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.269972 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-kube-api-access-mj88b" pod "c9670a7d-3aa9-4225-9cbb-783d6dcc4734" (UID: "c9670a7d-3aa9-4225-9cbb-783d6dcc4734"). InnerVolumeSpecName "kube-api-access-mj88b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:29:36.270058 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.269991 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-maas-api-tls" pod "c9670a7d-3aa9-4225-9cbb-783d6dcc4734" (UID: "c9670a7d-3aa9-4225-9cbb-783d6dcc4734"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:29:36.369552 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.369465 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:36.369552 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.369502 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:36.369772 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.369518 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj88b\" (UniqueName: \"kubernetes.io/projected/c9670a7d-3aa9-4225-9cbb-783d6dcc4734-kube-api-access-mj88b\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:29:36.837630 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.837591 5086 generic.go:356] "Generic (PLEG): container finished" podID="c9670a7d-3aa9-4225-9cbb-783d6dcc4734" containerID="72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc" exitCode=0 Nov 03 11:29:36.838082 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.837624 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" event={"ID":"c9670a7d-3aa9-4225-9cbb-783d6dcc4734","Type":"ContainerDied","Data":"72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc"} Nov 03 11:29:36.838082 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.837639 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" Nov 03 11:29:36.838082 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.837676 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8" event={"ID":"c9670a7d-3aa9-4225-9cbb-783d6dcc4734","Type":"ContainerDied","Data":"86d4fd6df83e137478f9aecc720c9d168591fabe9ae26c0522f1f28e0565381b"} Nov 03 11:29:36.838082 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.837698 5086 scope.go:122] "RemoveContainer" containerID="72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc" Nov 03 11:29:36.847981 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.847965 5086 scope.go:122] "RemoveContainer" containerID="72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc" Nov 03 11:29:36.848250 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:29:36.848233 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc\": container with ID starting with 72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc not found: ID does not exist" containerID="72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc" Nov 03 11:29:36.848341 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.848255 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc"} err="failed to get container status \"72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc\": rpc error: code = NotFound desc = could not find container \"72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc\": container with ID starting with 72d3899bb08de1fc00da0ded97448f94cd67ceb70a2a640c19ad794ec91ad5fc not found: ID does not exist" Nov 03 11:29:36.862960 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.862932 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8"] Nov 03 11:29:36.866797 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:36.866774 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-fcc2ce-696b5b4db7-v6qg8"] Nov 03 11:29:38.170557 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:38.170521 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9670a7d-3aa9-4225-9cbb-783d6dcc4734" path="/var/lib/kubelet/pods/c9670a7d-3aa9-4225-9cbb-783d6dcc4734/volumes" Nov 03 11:29:54.274222 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:29:54.274183 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:29:54.692805 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:54.692764 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng"] Nov 03 11:29:59.693426 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:29:59.693348 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" podUID="7124d01f-392b-490a-9688-3a4e3b9f50f1" containerName="payload-pre-processing" containerID="cri-o://71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f" gracePeriod=25 Nov 03 11:30:09.468361 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:09.468321 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" podUID="7124d01f-392b-490a-9688-3a4e3b9f50f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 11:30:15.384493 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.384453 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq"] Nov 03 11:30:15.384955 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.384668 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" podUID="d54b5afc-e80a-4c14-aafd-8ff9ad7da046" containerName="maas-api" containerID="cri-o://be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33" gracePeriod=30 Nov 03 11:30:15.734979 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.734399 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 11:30:15.822490 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.822455 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-kube-api-access-p89j5\" (UniqueName: \"kubernetes.io/projected/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-kube-api-access-p89j5\") pod \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " Nov 03 11:30:15.822683 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.822508 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-openshift-service-ca\") pod \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " Nov 03 11:30:15.822683 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.822543 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-metrics-certs\") pod \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " Nov 03 11:30:15.822828 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.822812 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-maas-api-tls\") pod \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\" (UID: \"d54b5afc-e80a-4c14-aafd-8ff9ad7da046\") " Nov 03 11:30:15.826490 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.826447 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-maas-api-tls" pod "d54b5afc-e80a-4c14-aafd-8ff9ad7da046" (UID: "d54b5afc-e80a-4c14-aafd-8ff9ad7da046"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:30:15.827523 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.827474 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-openshift-service-ca" pod "d54b5afc-e80a-4c14-aafd-8ff9ad7da046" (UID: "d54b5afc-e80a-4c14-aafd-8ff9ad7da046"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:30:15.828773 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.828743 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-metrics-certs" pod "d54b5afc-e80a-4c14-aafd-8ff9ad7da046" (UID: "d54b5afc-e80a-4c14-aafd-8ff9ad7da046"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:30:15.833757 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.833712 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-kube-api-access-p89j5" pod "d54b5afc-e80a-4c14-aafd-8ff9ad7da046" (UID: "d54b5afc-e80a-4c14-aafd-8ff9ad7da046"). InnerVolumeSpecName "kube-api-access-p89j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:30:15.926267 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.926229 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:15.926267 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.926268 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p89j5\" (UniqueName: \"kubernetes.io/projected/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-kube-api-access-p89j5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:15.926574 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.926284 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:15.926574 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:15.926312 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d54b5afc-e80a-4c14-aafd-8ff9ad7da046-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:16.118851 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.118817 5086 generic.go:356] "Generic (PLEG): container finished" podID="d54b5afc-e80a-4c14-aafd-8ff9ad7da046" containerID="be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33" exitCode=0 Nov 03 11:30:16.119078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.118879 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" Nov 03 11:30:16.119078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.118900 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" event={"ID":"d54b5afc-e80a-4c14-aafd-8ff9ad7da046","Type":"ContainerDied","Data":"be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33"} Nov 03 11:30:16.119078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.119006 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq" event={"ID":"d54b5afc-e80a-4c14-aafd-8ff9ad7da046","Type":"ContainerDied","Data":"8d2fe6aeac7f0acf2b0794e2a2092a85429f612ec6cfc450bc86247dba833be9"} Nov 03 11:30:16.119078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.119042 5086 scope.go:122] "RemoveContainer" containerID="be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33" Nov 03 11:30:16.133510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.133484 5086 scope.go:122] "RemoveContainer" containerID="be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33" Nov 03 11:30:16.133851 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:30:16.133829 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33\": container with ID starting with be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33 not found: ID does not exist" containerID="be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33" Nov 03 11:30:16.133942 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.133862 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33"} err="failed to get container status \"be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33\": rpc error: code = NotFound desc = could not find container \"be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33\": container with ID starting with be88c516d0f8a7cca9eb7137129f919ea32d9657939a1969c875ea2e375d1d33 not found: ID does not exist" Nov 03 11:30:16.144981 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.144946 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq"] Nov 03 11:30:16.152123 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.152078 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5d3c3-78d5dc8f77-dwdkq"] Nov 03 11:30:16.177590 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:16.177265 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54b5afc-e80a-4c14-aafd-8ff9ad7da046" path="/var/lib/kubelet/pods/d54b5afc-e80a-4c14-aafd-8ff9ad7da046/volumes" Nov 03 11:30:19.468303 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:19.468263 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" podUID="7124d01f-392b-490a-9688-3a4e3b9f50f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 11:30:19.962622 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:19.962592 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 11:30:19.990001 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:19.989970 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7124d01f-392b-490a-9688-3a4e3b9f50f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7124d01f-392b-490a-9688-3a4e3b9f50f1-plugins-config-volume\") pod \"7124d01f-392b-490a-9688-3a4e3b9f50f1\" (UID: \"7124d01f-392b-490a-9688-3a4e3b9f50f1\") " Nov 03 11:30:19.990160 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:19.990028 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7124d01f-392b-490a-9688-3a4e3b9f50f1-kube-api-access-52b7m\" (UniqueName: \"kubernetes.io/projected/7124d01f-392b-490a-9688-3a4e3b9f50f1-kube-api-access-52b7m\") pod \"7124d01f-392b-490a-9688-3a4e3b9f50f1\" (UID: \"7124d01f-392b-490a-9688-3a4e3b9f50f1\") " Nov 03 11:30:19.990411 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:19.990386 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7124d01f-392b-490a-9688-3a4e3b9f50f1-plugins-config-volume" pod "7124d01f-392b-490a-9688-3a4e3b9f50f1" (UID: "7124d01f-392b-490a-9688-3a4e3b9f50f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:30:19.992615 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:19.992585 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7124d01f-392b-490a-9688-3a4e3b9f50f1-kube-api-access-52b7m" pod "7124d01f-392b-490a-9688-3a4e3b9f50f1" (UID: "7124d01f-392b-490a-9688-3a4e3b9f50f1"). InnerVolumeSpecName "kube-api-access-52b7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:30:20.091727 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.091693 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7124d01f-392b-490a-9688-3a4e3b9f50f1-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:20.091727 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.091726 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52b7m\" (UniqueName: \"kubernetes.io/projected/7124d01f-392b-490a-9688-3a4e3b9f50f1-kube-api-access-52b7m\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:20.146119 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.146085 5086 generic.go:356] "Generic (PLEG): container finished" podID="7124d01f-392b-490a-9688-3a4e3b9f50f1" containerID="71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f" exitCode=0 Nov 03 11:30:20.146321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.146130 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" Nov 03 11:30:20.146321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.146175 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" event={"ID":"7124d01f-392b-490a-9688-3a4e3b9f50f1","Type":"ContainerDied","Data":"71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f"} Nov 03 11:30:20.146321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.146232 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng" event={"ID":"7124d01f-392b-490a-9688-3a4e3b9f50f1","Type":"ContainerDied","Data":"798f870da86c9fe8c70c2fd314ee5d5a2c73b039b6edf920c712c8b2b8de905c"} Nov 03 11:30:20.146321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.146248 5086 scope.go:122] "RemoveContainer" containerID="71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f" Nov 03 11:30:20.156383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.156368 5086 scope.go:122] "RemoveContainer" containerID="71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f" Nov 03 11:30:20.156647 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:30:20.156626 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f\": container with ID starting with 71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f not found: ID does not exist" containerID="71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f" Nov 03 11:30:20.156745 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.156651 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f"} err="failed to get container status \"71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f\": rpc error: code = NotFound desc = could not find container \"71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f\": container with ID starting with 71b6a201ff602670b74fc6d43986fe8dfc6deab6ce3f3deb8de17b6c8d43618f not found: ID does not exist" Nov 03 11:30:20.171759 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.171737 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng"] Nov 03 11:30:20.171866 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:20.171769 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d5d3c3-7c956d5fff-mgzng"] Nov 03 11:30:22.171436 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:22.171405 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7124d01f-392b-490a-9688-3a4e3b9f50f1" path="/var/lib/kubelet/pods/7124d01f-392b-490a-9688-3a4e3b9f50f1/volumes" Nov 03 11:30:22.615618 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:22.615577 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq"] Nov 03 11:30:22.616759 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:22.616538 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerName="istio-proxy" containerID="cri-o://4d745b22ef11aa17901fde13c8616b1e7a9af77f82ccac53950594fe9603e368" gracePeriod=30 Nov 03 11:30:24.274991 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:30:24.274955 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:30:24.975334 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:24.975280 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm"] Nov 03 11:30:24.975580 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:24.975552 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" podUID="ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" containerName="maas-api" containerID="cri-o://7bfdfe1e5c52f6f9ec96996630485d51854115ef8867be1654e4c374fccef9ac" gracePeriod=30 Nov 03 11:30:25.202203 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.202173 5086 generic.go:356] "Generic (PLEG): container finished" podID="ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" containerID="7bfdfe1e5c52f6f9ec96996630485d51854115ef8867be1654e4c374fccef9ac" exitCode=0 Nov 03 11:30:25.202640 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.202611 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" event={"ID":"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc","Type":"ContainerDied","Data":"7bfdfe1e5c52f6f9ec96996630485d51854115ef8867be1654e4c374fccef9ac"} Nov 03 11:30:25.283010 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.282973 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:30:25.388935 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.388900 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-kube-api-access-wz6pn\" (UniqueName: \"kubernetes.io/projected/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-kube-api-access-wz6pn\") pod \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " Nov 03 11:30:25.389095 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.389038 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-metrics-certs\") pod \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " Nov 03 11:30:25.389095 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.389065 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-openshift-service-ca\") pod \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " Nov 03 11:30:25.389172 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.389100 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls\") pod \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\" (UID: \"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc\") " Nov 03 11:30:25.389521 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.389492 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-openshift-service-ca" pod "ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" (UID: "ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:30:25.391531 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.391504 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls" pod "ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" (UID: "ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:30:25.391654 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.391575 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-metrics-certs" pod "ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" (UID: "ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:30:25.393126 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.393099 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-kube-api-access-wz6pn" pod "ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" (UID: "ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc"). InnerVolumeSpecName "kube-api-access-wz6pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:30:25.490612 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.490579 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:25.490612 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.490610 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:25.490805 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.490624 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:25.490805 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:25.490636 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz6pn\" (UniqueName: \"kubernetes.io/projected/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc-kube-api-access-wz6pn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:26.211115 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:26.211083 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" event={"ID":"ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc","Type":"ContainerDied","Data":"969e4acf31d329fa7dc906321d4be52800c3cf047fa9f1bb51554cde89a0a3a7"} Nov 03 11:30:26.211115 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:26.211115 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm" Nov 03 11:30:26.211367 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:26.211155 5086 scope.go:122] "RemoveContainer" containerID="7bfdfe1e5c52f6f9ec96996630485d51854115ef8867be1654e4c374fccef9ac" Nov 03 11:30:26.244563 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:26.244522 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm"] Nov 03 11:30:26.286500 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:26.286460 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-185908-5d6b699467-kfvzm"] Nov 03 11:30:28.172429 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.172397 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" path="/var/lib/kubelet/pods/ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc/volumes" Nov 03 11:30:28.239342 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.231474 5086 generic.go:356] "Generic (PLEG): container finished" podID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerID="4d745b22ef11aa17901fde13c8616b1e7a9af77f82ccac53950594fe9603e368" exitCode=0 Nov 03 11:30:28.239342 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.231605 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" event={"ID":"c4973c07-e22e-4d8a-be06-505b9d3b7297","Type":"ContainerDied","Data":"4d745b22ef11aa17901fde13c8616b1e7a9af77f82ccac53950594fe9603e368"} Nov 03 11:30:28.301432 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.301409 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 11:30:28.430954 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.430867 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-kube-api-access-9gpbm\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-kube-api-access-9gpbm\") pod \"c4973c07-e22e-4d8a-be06-505b9d3b7297\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " Nov 03 11:30:28.430954 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.430950 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-credential-socket\") pod \"c4973c07-e22e-4d8a-be06-505b9d3b7297\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " Nov 03 11:30:28.431161 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431015 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-certs\") pod \"c4973c07-e22e-4d8a-be06-505b9d3b7297\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " Nov 03 11:30:28.431161 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431037 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-podinfo\") pod \"c4973c07-e22e-4d8a-be06-505b9d3b7297\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " Nov 03 11:30:28.431161 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431088 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-envoy\") pod \"c4973c07-e22e-4d8a-be06-505b9d3b7297\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " Nov 03 11:30:28.431161 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431115 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-data\") pod \"c4973c07-e22e-4d8a-be06-505b9d3b7297\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " Nov 03 11:30:28.431443 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431187 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4973c07-e22e-4d8a-be06-505b9d3b7297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4973c07-e22e-4d8a-be06-505b9d3b7297-istiod-ca-cert\") pod \"c4973c07-e22e-4d8a-be06-505b9d3b7297\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " Nov 03 11:30:28.431443 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431226 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-socket\") pod \"c4973c07-e22e-4d8a-be06-505b9d3b7297\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " Nov 03 11:30:28.431443 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431277 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-token\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-token\") pod \"c4973c07-e22e-4d8a-be06-505b9d3b7297\" (UID: \"c4973c07-e22e-4d8a-be06-505b9d3b7297\") " Nov 03 11:30:28.431443 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431277 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-credential-socket" pod "c4973c07-e22e-4d8a-be06-505b9d3b7297" (UID: "c4973c07-e22e-4d8a-be06-505b9d3b7297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:30:28.431443 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431331 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-certs" pod "c4973c07-e22e-4d8a-be06-505b9d3b7297" (UID: "c4973c07-e22e-4d8a-be06-505b9d3b7297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:30:28.431722 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431499 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-data" pod "c4973c07-e22e-4d8a-be06-505b9d3b7297" (UID: "c4973c07-e22e-4d8a-be06-505b9d3b7297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:30:28.431722 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431588 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-socket" pod "c4973c07-e22e-4d8a-be06-505b9d3b7297" (UID: "c4973c07-e22e-4d8a-be06-505b9d3b7297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:30:28.431722 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431666 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4973c07-e22e-4d8a-be06-505b9d3b7297-istiod-ca-cert" pod "c4973c07-e22e-4d8a-be06-505b9d3b7297" (UID: "c4973c07-e22e-4d8a-be06-505b9d3b7297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:30:28.431921 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431897 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:28.431921 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431923 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:28.432133 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431940 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4973c07-e22e-4d8a-be06-505b9d3b7297-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:28.432133 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431957 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:28.432133 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.431970 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:28.433646 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.433615 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-envoy" pod "c4973c07-e22e-4d8a-be06-505b9d3b7297" (UID: "c4973c07-e22e-4d8a-be06-505b9d3b7297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:30:28.433760 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.433739 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-kube-api-access-9gpbm" pod "c4973c07-e22e-4d8a-be06-505b9d3b7297" (UID: "c4973c07-e22e-4d8a-be06-505b9d3b7297"). InnerVolumeSpecName "kube-api-access-9gpbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:30:28.433847 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.433831 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-podinfo" pod "c4973c07-e22e-4d8a-be06-505b9d3b7297" (UID: "c4973c07-e22e-4d8a-be06-505b9d3b7297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:30:28.434060 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.434043 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-token" pod "c4973c07-e22e-4d8a-be06-505b9d3b7297" (UID: "c4973c07-e22e-4d8a-be06-505b9d3b7297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:30:28.532819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.532787 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:28.532819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.532812 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gpbm\" (UniqueName: \"kubernetes.io/projected/c4973c07-e22e-4d8a-be06-505b9d3b7297-kube-api-access-9gpbm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:28.532819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.532822 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:28.532819 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:28.532831 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4973c07-e22e-4d8a-be06-505b9d3b7297-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:29.239943 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:29.239914 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" Nov 03 11:30:29.240448 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:29.239917 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq" event={"ID":"c4973c07-e22e-4d8a-be06-505b9d3b7297","Type":"ContainerDied","Data":"a4be66d17f0ac18258177401ed0f200ef68961ba7c7fa0ba9f0afdc747db7ea1"} Nov 03 11:30:29.240448 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:29.240041 5086 scope.go:122] "RemoveContainer" containerID="4d745b22ef11aa17901fde13c8616b1e7a9af77f82ccac53950594fe9603e368" Nov 03 11:30:29.264546 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:29.264511 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq"] Nov 03 11:30:29.269483 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:29.269455 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-j9qwq"] Nov 03 11:30:30.170653 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:30.170618 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" path="/var/lib/kubelet/pods/c4973c07-e22e-4d8a-be06-505b9d3b7297/volumes" Nov 03 11:30:47.929634 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:47.929599 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g"] Nov 03 11:30:47.930891 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:47.930853 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" podUID="482a443b-f6da-4644-a6ae-198e22602468" containerName="maas-api" containerID="cri-o://1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a" gracePeriod=30 Nov 03 11:30:48.205612 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.205583 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:30:48.262215 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.262181 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482a443b-f6da-4644-a6ae-198e22602468-kube-api-access-hn5xz\" (UniqueName: \"kubernetes.io/projected/482a443b-f6da-4644-a6ae-198e22602468-kube-api-access-hn5xz\") pod \"482a443b-f6da-4644-a6ae-198e22602468\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " Nov 03 11:30:48.262451 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.262243 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-metrics-certs\") pod \"482a443b-f6da-4644-a6ae-198e22602468\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " Nov 03 11:30:48.262451 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.262331 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-maas-api-tls\") pod \"482a443b-f6da-4644-a6ae-198e22602468\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " Nov 03 11:30:48.262601 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.262487 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482a443b-f6da-4644-a6ae-198e22602468-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a443b-f6da-4644-a6ae-198e22602468-openshift-service-ca\") pod \"482a443b-f6da-4644-a6ae-198e22602468\" (UID: \"482a443b-f6da-4644-a6ae-198e22602468\") " Nov 03 11:30:48.262903 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.262874 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482a443b-f6da-4644-a6ae-198e22602468-openshift-service-ca" pod "482a443b-f6da-4644-a6ae-198e22602468" (UID: "482a443b-f6da-4644-a6ae-198e22602468"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:30:48.263098 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.263043 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a443b-f6da-4644-a6ae-198e22602468-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:48.264595 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.264566 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-maas-api-tls" pod "482a443b-f6da-4644-a6ae-198e22602468" (UID: "482a443b-f6da-4644-a6ae-198e22602468"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:30:48.264708 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.264586 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482a443b-f6da-4644-a6ae-198e22602468-kube-api-access-hn5xz" pod "482a443b-f6da-4644-a6ae-198e22602468" (UID: "482a443b-f6da-4644-a6ae-198e22602468"). InnerVolumeSpecName "kube-api-access-hn5xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:30:48.264780 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.264731 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-metrics-certs" pod "482a443b-f6da-4644-a6ae-198e22602468" (UID: "482a443b-f6da-4644-a6ae-198e22602468"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:30:48.364332 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.364255 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5xz\" (UniqueName: \"kubernetes.io/projected/482a443b-f6da-4644-a6ae-198e22602468-kube-api-access-hn5xz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:48.364332 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.364330 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:48.364564 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.364348 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a443b-f6da-4644-a6ae-198e22602468-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:30:48.373995 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.373960 5086 generic.go:356] "Generic (PLEG): container finished" podID="482a443b-f6da-4644-a6ae-198e22602468" containerID="1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a" exitCode=0 Nov 03 11:30:48.374176 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.374062 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" event={"ID":"482a443b-f6da-4644-a6ae-198e22602468","Type":"ContainerDied","Data":"1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a"} Nov 03 11:30:48.374176 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.374114 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" event={"ID":"482a443b-f6da-4644-a6ae-198e22602468","Type":"ContainerDied","Data":"331d7966ceea7cbc8bf465b99a497671aa98c9f42d125ef7ea01375d826dcf2f"} Nov 03 11:30:48.374176 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.374141 5086 scope.go:122] "RemoveContainer" containerID="1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a" Nov 03 11:30:48.374383 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.374359 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g" Nov 03 11:30:48.392196 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.392158 5086 scope.go:122] "RemoveContainer" containerID="1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a" Nov 03 11:30:48.392608 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:30:48.392582 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a\": container with ID starting with 1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a not found: ID does not exist" containerID="1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a" Nov 03 11:30:48.392707 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.392622 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a"} err="failed to get container status \"1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a\": rpc error: code = NotFound desc = could not find container \"1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a\": container with ID starting with 1b1008293837af33fbdefa93beb2dbde8785d1aa37055d07a9a0f210d7f88f4a not found: ID does not exist" Nov 03 11:30:48.409853 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.409804 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g"] Nov 03 11:30:48.413907 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:48.413881 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56687c-84687d5684-xpv9g"] Nov 03 11:30:50.170965 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:30:50.170919 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482a443b-f6da-4644-a6ae-198e22602468" path="/var/lib/kubelet/pods/482a443b-f6da-4644-a6ae-198e22602468/volumes" Nov 03 11:30:54.273796 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:30:54.273715 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:31:15.159413 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.159376 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl"] Nov 03 11:31:15.159979 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.159656 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" podUID="d277c8b6-7753-4614-959d-e79d38a77fc4" containerName="maas-api" containerID="cri-o://bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077" gracePeriod=30 Nov 03 11:31:15.479729 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.479702 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 11:31:15.515700 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.515654 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d277c8b6-7753-4614-959d-e79d38a77fc4-kube-api-access-sjqgv\" (UniqueName: \"kubernetes.io/projected/d277c8b6-7753-4614-959d-e79d38a77fc4-kube-api-access-sjqgv\") pod \"d277c8b6-7753-4614-959d-e79d38a77fc4\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " Nov 03 11:31:15.515898 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.515744 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-metrics-certs\") pod \"d277c8b6-7753-4614-959d-e79d38a77fc4\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " Nov 03 11:31:15.515898 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.515865 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d277c8b6-7753-4614-959d-e79d38a77fc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d277c8b6-7753-4614-959d-e79d38a77fc4-openshift-service-ca\") pod \"d277c8b6-7753-4614-959d-e79d38a77fc4\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " Nov 03 11:31:15.516965 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.516009 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls\") pod \"d277c8b6-7753-4614-959d-e79d38a77fc4\" (UID: \"d277c8b6-7753-4614-959d-e79d38a77fc4\") " Nov 03 11:31:15.518376 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.518344 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d277c8b6-7753-4614-959d-e79d38a77fc4-openshift-service-ca" pod "d277c8b6-7753-4614-959d-e79d38a77fc4" (UID: "d277c8b6-7753-4614-959d-e79d38a77fc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:31:15.524045 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.522158 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d277c8b6-7753-4614-959d-e79d38a77fc4-kube-api-access-sjqgv" pod "d277c8b6-7753-4614-959d-e79d38a77fc4" (UID: "d277c8b6-7753-4614-959d-e79d38a77fc4"). InnerVolumeSpecName "kube-api-access-sjqgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:31:15.524707 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.524674 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls" pod "d277c8b6-7753-4614-959d-e79d38a77fc4" (UID: "d277c8b6-7753-4614-959d-e79d38a77fc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:31:15.525433 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.525403 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-metrics-certs" pod "d277c8b6-7753-4614-959d-e79d38a77fc4" (UID: "d277c8b6-7753-4614-959d-e79d38a77fc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:31:15.549401 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.549372 5086 generic.go:356] "Generic (PLEG): container finished" podID="d277c8b6-7753-4614-959d-e79d38a77fc4" containerID="bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077" exitCode=0 Nov 03 11:31:15.549551 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.549416 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" Nov 03 11:31:15.549551 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.549458 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" event={"ID":"d277c8b6-7753-4614-959d-e79d38a77fc4","Type":"ContainerDied","Data":"bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077"} Nov 03 11:31:15.549551 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.549516 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl" event={"ID":"d277c8b6-7753-4614-959d-e79d38a77fc4","Type":"ContainerDied","Data":"9207cfc37fff402bcdb8c66b4015d7837a50634fcdfcb87e824194441b82e47c"} Nov 03 11:31:15.549551 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.549539 5086 scope.go:122] "RemoveContainer" containerID="bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077" Nov 03 11:31:15.560981 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.560966 5086 scope.go:122] "RemoveContainer" containerID="bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077" Nov 03 11:31:15.561266 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:31:15.561248 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077\": container with ID starting with bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077 not found: ID does not exist" containerID="bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077" Nov 03 11:31:15.561325 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.561274 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077"} err="failed to get container status \"bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077\": rpc error: code = NotFound desc = could not find container \"bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077\": container with ID starting with bacd0e80766229cab8e0957782d65708b5739e3d3c9daa98a2a469b089881077 not found: ID does not exist" Nov 03 11:31:15.572976 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.572951 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl"] Nov 03 11:31:15.576692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.576668 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9c53b2-87568c868-wpztl"] Nov 03 11:31:15.619920 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.619884 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d277c8b6-7753-4614-959d-e79d38a77fc4-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:15.619920 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.619922 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:15.620217 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.619940 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjqgv\" (UniqueName: \"kubernetes.io/projected/d277c8b6-7753-4614-959d-e79d38a77fc4-kube-api-access-sjqgv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:15.620217 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:15.619954 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d277c8b6-7753-4614-959d-e79d38a77fc4-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:16.171053 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:16.171023 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d277c8b6-7753-4614-959d-e79d38a77fc4" path="/var/lib/kubelet/pods/d277c8b6-7753-4614-959d-e79d38a77fc4/volumes" Nov 03 11:31:18.273944 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.273909 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9"] Nov 03 11:31:18.274469 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.274168 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" podUID="bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" containerName="maas-api" containerID="cri-o://9021f2fb00782ed847a1596a9e1a08e5c44fa74362f7c98257fbf19277b47c9f" gracePeriod=30 Nov 03 11:31:18.581219 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.580337 5086 generic.go:356] "Generic (PLEG): container finished" podID="bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" containerID="9021f2fb00782ed847a1596a9e1a08e5c44fa74362f7c98257fbf19277b47c9f" exitCode=0 Nov 03 11:31:18.581219 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.580409 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" event={"ID":"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91","Type":"ContainerDied","Data":"9021f2fb00782ed847a1596a9e1a08e5c44fa74362f7c98257fbf19277b47c9f"} Nov 03 11:31:18.581219 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.580457 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" event={"ID":"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91","Type":"ContainerDied","Data":"25e4add8040e834637373038f0d7b61d20e53f3386051c3d3dbb44c3cc5728d7"} Nov 03 11:31:18.581219 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.580484 5086 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e4add8040e834637373038f0d7b61d20e53f3386051c3d3dbb44c3cc5728d7" Nov 03 11:31:18.594164 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.594137 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:31:18.661496 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.661462 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-openshift-service-ca\") pod \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " Nov 03 11:31:18.661700 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.661591 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-kube-api-access-mxs8r\" (UniqueName: \"kubernetes.io/projected/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-kube-api-access-mxs8r\") pod \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " Nov 03 11:31:18.661774 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.661698 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-maas-api-tls\") pod \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " Nov 03 11:31:18.661774 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.661737 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-metrics-certs\") pod \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\" (UID: \"bf8ec1bc-f67d-4d10-836c-7fa857ea9a91\") " Nov 03 11:31:18.661921 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.661899 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-openshift-service-ca" pod "bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" (UID: "bf8ec1bc-f67d-4d10-836c-7fa857ea9a91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:31:18.662165 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.662146 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:18.663781 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.663747 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-metrics-certs" pod "bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" (UID: "bf8ec1bc-f67d-4d10-836c-7fa857ea9a91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:31:18.663781 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.663752 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-kube-api-access-mxs8r" pod "bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" (UID: "bf8ec1bc-f67d-4d10-836c-7fa857ea9a91"). InnerVolumeSpecName "kube-api-access-mxs8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:31:18.663952 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.663857 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-maas-api-tls" pod "bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" (UID: "bf8ec1bc-f67d-4d10-836c-7fa857ea9a91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:31:18.763612 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.763579 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxs8r\" (UniqueName: \"kubernetes.io/projected/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-kube-api-access-mxs8r\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:18.763612 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.763611 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:18.763612 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:18.763621 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:19.586209 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:19.586178 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9" Nov 03 11:31:19.612527 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:19.612499 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9"] Nov 03 11:31:19.616159 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:19.616125 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-917da2-b6bbd5b98-dbkr9"] Nov 03 11:31:20.171439 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:20.171395 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" path="/var/lib/kubelet/pods/bf8ec1bc-f67d-4d10-836c-7fa857ea9a91/volumes" Nov 03 11:31:24.274232 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:31:24.274195 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:31:38.939030 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:38.938990 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9"] Nov 03 11:31:38.939633 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:38.939256 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" podUID="0063172a-cdd0-4c8d-b0c9-33f77c87c769" containerName="maas-api" containerID="cri-o://800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c" gracePeriod=30 Nov 03 11:31:39.204499 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.204442 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 11:31:39.302785 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.302752 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-metrics-certs\") pod \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " Nov 03 11:31:39.303013 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.302811 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls\") pod \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " Nov 03 11:31:39.303013 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.302884 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0063172a-cdd0-4c8d-b0c9-33f77c87c769-kube-api-access-8ptsb\" (UniqueName: \"kubernetes.io/projected/0063172a-cdd0-4c8d-b0c9-33f77c87c769-kube-api-access-8ptsb\") pod \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " Nov 03 11:31:39.303013 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.302944 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0063172a-cdd0-4c8d-b0c9-33f77c87c769-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063172a-cdd0-4c8d-b0c9-33f77c87c769-openshift-service-ca\") pod \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\" (UID: \"0063172a-cdd0-4c8d-b0c9-33f77c87c769\") " Nov 03 11:31:39.303503 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.303476 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0063172a-cdd0-4c8d-b0c9-33f77c87c769-openshift-service-ca" pod "0063172a-cdd0-4c8d-b0c9-33f77c87c769" (UID: "0063172a-cdd0-4c8d-b0c9-33f77c87c769"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:31:39.305560 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.305528 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0063172a-cdd0-4c8d-b0c9-33f77c87c769-kube-api-access-8ptsb" pod "0063172a-cdd0-4c8d-b0c9-33f77c87c769" (UID: "0063172a-cdd0-4c8d-b0c9-33f77c87c769"). InnerVolumeSpecName "kube-api-access-8ptsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:31:39.307734 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.307710 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls" pod "0063172a-cdd0-4c8d-b0c9-33f77c87c769" (UID: "0063172a-cdd0-4c8d-b0c9-33f77c87c769"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:31:39.309423 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.309400 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-metrics-certs" pod "0063172a-cdd0-4c8d-b0c9-33f77c87c769" (UID: "0063172a-cdd0-4c8d-b0c9-33f77c87c769"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:31:39.404139 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.404091 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:39.404139 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.404138 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ptsb\" (UniqueName: \"kubernetes.io/projected/0063172a-cdd0-4c8d-b0c9-33f77c87c769-kube-api-access-8ptsb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:39.404360 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.404152 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063172a-cdd0-4c8d-b0c9-33f77c87c769-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:39.404360 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.404161 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0063172a-cdd0-4c8d-b0c9-33f77c87c769-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:39.715457 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.715420 5086 generic.go:356] "Generic (PLEG): container finished" podID="0063172a-cdd0-4c8d-b0c9-33f77c87c769" containerID="800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c" exitCode=0 Nov 03 11:31:39.715674 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.715475 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" Nov 03 11:31:39.715674 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.715513 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" event={"ID":"0063172a-cdd0-4c8d-b0c9-33f77c87c769","Type":"ContainerDied","Data":"800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c"} Nov 03 11:31:39.715674 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.715579 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9" event={"ID":"0063172a-cdd0-4c8d-b0c9-33f77c87c769","Type":"ContainerDied","Data":"d4944b890fa85a12e9bcecbe08add095e1786ac20b42af46ecdfc14f87f65db2"} Nov 03 11:31:39.715674 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.715613 5086 scope.go:122] "RemoveContainer" containerID="800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c" Nov 03 11:31:39.726503 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.726487 5086 scope.go:122] "RemoveContainer" containerID="800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c" Nov 03 11:31:39.726756 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:31:39.726738 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c\": container with ID starting with 800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c not found: ID does not exist" containerID="800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c" Nov 03 11:31:39.726804 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.726764 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c"} err="failed to get container status \"800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c\": rpc error: code = NotFound desc = could not find container \"800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c\": container with ID starting with 800eb28d8479d668aef60036fb17ebbda38113c98739ccc659a67d234ccfa37c not found: ID does not exist" Nov 03 11:31:39.741640 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.741619 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9"] Nov 03 11:31:39.745308 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:39.745276 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ef1662-54459885b5-s4wf9"] Nov 03 11:31:40.184355 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:40.181689 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0063172a-cdd0-4c8d-b0c9-33f77c87c769" path="/var/lib/kubelet/pods/0063172a-cdd0-4c8d-b0c9-33f77c87c769/volumes" Nov 03 11:31:44.995844 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:44.995809 5086 scope.go:122] "RemoveContainer" containerID="511ce26580d4ea6ce4344066c78979225b8db97a8fcd119d88f3c19513b01b65" Nov 03 11:31:45.010422 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:45.010396 5086 scope.go:122] "RemoveContainer" containerID="9021f2fb00782ed847a1596a9e1a08e5c44fa74362f7c98257fbf19277b47c9f" Nov 03 11:31:47.633402 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.633364 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph"] Nov 03 11:31:47.633925 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.633653 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" podUID="369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" containerName="maas-api" containerID="cri-o://da86201f51df02ea5401dcc82e9acf46a2f98ac18b49e40b3cfb1750a4721ff3" gracePeriod=30 Nov 03 11:31:47.773954 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.773918 5086 generic.go:356] "Generic (PLEG): container finished" podID="369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" containerID="da86201f51df02ea5401dcc82e9acf46a2f98ac18b49e40b3cfb1750a4721ff3" exitCode=0 Nov 03 11:31:47.774117 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.773996 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" event={"ID":"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a","Type":"ContainerDied","Data":"da86201f51df02ea5401dcc82e9acf46a2f98ac18b49e40b3cfb1750a4721ff3"} Nov 03 11:31:47.948718 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.948693 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:31:48.001076 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.998672 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-openshift-service-ca\") pod \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " Nov 03 11:31:48.001076 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.998836 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-kube-api-access-td7pf\" (UniqueName: \"kubernetes.io/projected/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-kube-api-access-td7pf\") pod \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " Nov 03 11:31:48.001076 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.998968 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-metrics-certs\") pod \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " Nov 03 11:31:48.001076 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.999069 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls\") pod \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\" (UID: \"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a\") " Nov 03 11:31:48.001076 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:47.999580 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-openshift-service-ca" pod "369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" (UID: "369e9ba2-4a9d-4a48-9fa3-7a10ad04803a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:31:48.001924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.001897 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-kube-api-access-td7pf" pod "369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" (UID: "369e9ba2-4a9d-4a48-9fa3-7a10ad04803a"). InnerVolumeSpecName "kube-api-access-td7pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:31:48.001924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.002236 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-metrics-certs" pod "369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" (UID: "369e9ba2-4a9d-4a48-9fa3-7a10ad04803a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:31:48.001924 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.002382 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls" pod "369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" (UID: "369e9ba2-4a9d-4a48-9fa3-7a10ad04803a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:31:48.100860 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.100827 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:48.100860 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.100858 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:48.101128 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.100871 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td7pf\" (UniqueName: \"kubernetes.io/projected/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-kube-api-access-td7pf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:48.101128 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.100886 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:31:48.781750 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.781719 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" Nov 03 11:31:48.782218 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.781758 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph" event={"ID":"369e9ba2-4a9d-4a48-9fa3-7a10ad04803a","Type":"ContainerDied","Data":"faf32b9055adbbb7a213909fcd92fca0d107579bf96eb7130e947e156b1e55da"} Nov 03 11:31:48.782218 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.781834 5086 scope.go:122] "RemoveContainer" containerID="da86201f51df02ea5401dcc82e9acf46a2f98ac18b49e40b3cfb1750a4721ff3" Nov 03 11:31:48.798823 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.798799 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph"] Nov 03 11:31:48.802614 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:48.802589 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb5766-fbd976fd7-5xpph"] Nov 03 11:31:50.171229 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:50.171196 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" path="/var/lib/kubelet/pods/369e9ba2-4a9d-4a48-9fa3-7a10ad04803a/volumes" Nov 03 11:31:51.952217 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:31:51.952176 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf"] Nov 03 11:31:54.274642 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:31:54.274606 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:32:05.841655 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:05.841619 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b"] Nov 03 11:32:06.029628 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:06.029570 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v"] Nov 03 11:32:06.974102 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:06.974048 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" containerID="cri-o://91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204" gracePeriod=45 Nov 03 11:32:06.997812 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:06.997778 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Nov 03 11:32:07.252288 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.252262 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 11:32:07.358710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.358676 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf375e6-6ba5-4fca-b377-72060451aa71-tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf375e6-6ba5-4fca-b377-72060451aa71-tls-certs\") pod \"ddf375e6-6ba5-4fca-b377-72060451aa71\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " Nov 03 11:32:07.358710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.358712 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf375e6-6ba5-4fca-b377-72060451aa71-kube-api-access-99psf\" (UniqueName: \"kubernetes.io/projected/ddf375e6-6ba5-4fca-b377-72060451aa71-kube-api-access-99psf\") pod \"ddf375e6-6ba5-4fca-b377-72060451aa71\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " Nov 03 11:32:07.358969 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.358749 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-tmp-dir\") pod \"ddf375e6-6ba5-4fca-b377-72060451aa71\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " Nov 03 11:32:07.358969 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.358763 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-home\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-home\") pod \"ddf375e6-6ba5-4fca-b377-72060451aa71\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " Nov 03 11:32:07.358969 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.358813 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-model-cache\") pod \"ddf375e6-6ba5-4fca-b377-72060451aa71\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " Nov 03 11:32:07.358969 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.358841 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-dshm\") pod \"ddf375e6-6ba5-4fca-b377-72060451aa71\" (UID: \"ddf375e6-6ba5-4fca-b377-72060451aa71\") " Nov 03 11:32:07.359200 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.359083 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-home" pod "ddf375e6-6ba5-4fca-b377-72060451aa71" (UID: "ddf375e6-6ba5-4fca-b377-72060451aa71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:32:07.359200 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.359060 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-tmp-dir" pod "ddf375e6-6ba5-4fca-b377-72060451aa71" (UID: "ddf375e6-6ba5-4fca-b377-72060451aa71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:32:07.359200 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.359108 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-model-cache" pod "ddf375e6-6ba5-4fca-b377-72060451aa71" (UID: "ddf375e6-6ba5-4fca-b377-72060451aa71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:32:07.359440 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.359423 5086 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:07.359510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.359442 5086 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:07.359510 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.359455 5086 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:07.361102 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.361076 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-dshm" pod "ddf375e6-6ba5-4fca-b377-72060451aa71" (UID: "ddf375e6-6ba5-4fca-b377-72060451aa71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:32:07.361214 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.361118 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf375e6-6ba5-4fca-b377-72060451aa71-tls-certs" pod "ddf375e6-6ba5-4fca-b377-72060451aa71" (UID: "ddf375e6-6ba5-4fca-b377-72060451aa71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:07.361214 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.361173 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf375e6-6ba5-4fca-b377-72060451aa71-kube-api-access-99psf" pod "ddf375e6-6ba5-4fca-b377-72060451aa71" (UID: "ddf375e6-6ba5-4fca-b377-72060451aa71"). InnerVolumeSpecName "kube-api-access-99psf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:32:07.460257 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.460216 5086 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf375e6-6ba5-4fca-b377-72060451aa71-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:07.460257 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.460251 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99psf\" (UniqueName: \"kubernetes.io/projected/ddf375e6-6ba5-4fca-b377-72060451aa71-kube-api-access-99psf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:07.460499 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.460267 5086 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf375e6-6ba5-4fca-b377-72060451aa71-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:07.902413 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.902376 5086 generic.go:356] "Generic (PLEG): container finished" podID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerID="91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204" exitCode=0 Nov 03 11:32:07.902597 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.902429 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" Nov 03 11:32:07.902597 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.902458 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" event={"ID":"ddf375e6-6ba5-4fca-b377-72060451aa71","Type":"ContainerDied","Data":"91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204"} Nov 03 11:32:07.902597 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.902505 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf" event={"ID":"ddf375e6-6ba5-4fca-b377-72060451aa71","Type":"ContainerDied","Data":"320d7c0cdf7c11e7cab91e2e881db6cde4387058a748e920e3806f773cbcc3d2"} Nov 03 11:32:07.902597 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.902521 5086 scope.go:122] "RemoveContainer" containerID="91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204" Nov 03 11:32:07.912503 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.912484 5086 scope.go:122] "RemoveContainer" containerID="91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204" Nov 03 11:32:07.912773 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:32:07.912753 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204\": container with ID starting with 91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204 not found: ID does not exist" containerID="91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204" Nov 03 11:32:07.912848 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.912784 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204"} err="failed to get container status \"91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204\": rpc error: code = NotFound desc = could not find container \"91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204\": container with ID starting with 91062d10b46e22af39660e0bc1f55eaf0ba47a4a8aa8d113718b7bcff8e34204 not found: ID does not exist" Nov 03 11:32:07.942693 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.942666 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf"] Nov 03 11:32:07.952364 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:07.952340 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/simulator-w0-f78621-kserve-7f98bbcc9-blwmf"] Nov 03 11:32:08.170920 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.170848 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" path="/var/lib/kubelet/pods/ddf375e6-6ba5-4fca-b377-72060451aa71/volumes" Nov 03 11:32:08.307591 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.307555 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47"] Nov 03 11:32:08.307881 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.307853 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" podUID="fd727856-e40a-4dc5-9c67-f44e8f89e123" containerName="maas-api" containerID="cri-o://9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6" gracePeriod=30 Nov 03 11:32:08.630801 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.630769 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 11:32:08.673693 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.673617 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-metrics-certs\") pod \"fd727856-e40a-4dc5-9c67-f44e8f89e123\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " Nov 03 11:32:08.673693 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.673657 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls\") pod \"fd727856-e40a-4dc5-9c67-f44e8f89e123\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " Nov 03 11:32:08.673693 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.673690 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd727856-e40a-4dc5-9c67-f44e8f89e123-kube-api-access-fbdng\" (UniqueName: \"kubernetes.io/projected/fd727856-e40a-4dc5-9c67-f44e8f89e123-kube-api-access-fbdng\") pod \"fd727856-e40a-4dc5-9c67-f44e8f89e123\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " Nov 03 11:32:08.674056 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.673773 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd727856-e40a-4dc5-9c67-f44e8f89e123-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd727856-e40a-4dc5-9c67-f44e8f89e123-openshift-service-ca\") pod \"fd727856-e40a-4dc5-9c67-f44e8f89e123\" (UID: \"fd727856-e40a-4dc5-9c67-f44e8f89e123\") " Nov 03 11:32:08.674227 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.674203 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd727856-e40a-4dc5-9c67-f44e8f89e123-openshift-service-ca" pod "fd727856-e40a-4dc5-9c67-f44e8f89e123" (UID: "fd727856-e40a-4dc5-9c67-f44e8f89e123"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:32:08.675986 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.675954 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-metrics-certs" pod "fd727856-e40a-4dc5-9c67-f44e8f89e123" (UID: "fd727856-e40a-4dc5-9c67-f44e8f89e123"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:08.676735 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.676706 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls" pod "fd727856-e40a-4dc5-9c67-f44e8f89e123" (UID: "fd727856-e40a-4dc5-9c67-f44e8f89e123"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:08.676821 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.676763 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd727856-e40a-4dc5-9c67-f44e8f89e123-kube-api-access-fbdng" pod "fd727856-e40a-4dc5-9c67-f44e8f89e123" (UID: "fd727856-e40a-4dc5-9c67-f44e8f89e123"). InnerVolumeSpecName "kube-api-access-fbdng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:32:08.775780 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.775742 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd727856-e40a-4dc5-9c67-f44e8f89e123-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:08.775780 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.775781 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:08.776059 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.775795 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd727856-e40a-4dc5-9c67-f44e8f89e123-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:08.776059 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.775810 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbdng\" (UniqueName: \"kubernetes.io/projected/fd727856-e40a-4dc5-9c67-f44e8f89e123-kube-api-access-fbdng\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:08.910759 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.910719 5086 generic.go:356] "Generic (PLEG): container finished" podID="fd727856-e40a-4dc5-9c67-f44e8f89e123" containerID="9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6" exitCode=0 Nov 03 11:32:08.910936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.910767 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" Nov 03 11:32:08.910936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.910797 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" event={"ID":"fd727856-e40a-4dc5-9c67-f44e8f89e123","Type":"ContainerDied","Data":"9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6"} Nov 03 11:32:08.910936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.910852 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47" event={"ID":"fd727856-e40a-4dc5-9c67-f44e8f89e123","Type":"ContainerDied","Data":"4870334d3a3f8b5cb628cb375e79c24b6b4b39b52b823759565cf821fee186db"} Nov 03 11:32:08.910936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.910874 5086 scope.go:122] "RemoveContainer" containerID="9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6" Nov 03 11:32:08.921315 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.921283 5086 scope.go:122] "RemoveContainer" containerID="9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6" Nov 03 11:32:08.921610 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:32:08.921591 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6\": container with ID starting with 9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6 not found: ID does not exist" containerID="9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6" Nov 03 11:32:08.921662 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.921618 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6"} err="failed to get container status \"9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6\": rpc error: code = NotFound desc = could not find container \"9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6\": container with ID starting with 9167dcaa68fd32f1a6d6f913f4923c921bc298539824bca9e5d28c98ef14bfe6 not found: ID does not exist" Nov 03 11:32:08.932905 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.932851 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47"] Nov 03 11:32:08.936287 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:08.936262 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6012b5-7cbb4b5df7-hfg47"] Nov 03 11:32:10.170325 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:10.170281 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd727856-e40a-4dc5-9c67-f44e8f89e123" path="/var/lib/kubelet/pods/fd727856-e40a-4dc5-9c67-f44e8f89e123/volumes" Nov 03 11:32:10.842397 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:10.842346 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" podUID="c1228536-409d-41c3-aab5-b70ea3c1f4e6" containerName="payload-pre-processing" containerID="cri-o://48b0f73f069fc392f03a44cb89ff41b45a99a274c407882c38df58ef849dfe82" gracePeriod=25 Nov 03 11:32:14.180356 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.179997 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r"] Nov 03 11:32:14.180356 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.180249 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" podUID="ac3318e5-f88a-40e6-9482-55a0e47c162e" containerName="maas-api" containerID="cri-o://857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366" gracePeriod=30 Nov 03 11:32:14.500678 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.500597 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 11:32:14.673180 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.672717 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3318e5-f88a-40e6-9482-55a0e47c162e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3318e5-f88a-40e6-9482-55a0e47c162e-openshift-service-ca\") pod \"ac3318e5-f88a-40e6-9482-55a0e47c162e\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " Nov 03 11:32:14.673180 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.672795 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-maas-api-tls\") pod \"ac3318e5-f88a-40e6-9482-55a0e47c162e\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " Nov 03 11:32:14.673180 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.672840 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3318e5-f88a-40e6-9482-55a0e47c162e-kube-api-access-nr7jt\" (UniqueName: \"kubernetes.io/projected/ac3318e5-f88a-40e6-9482-55a0e47c162e-kube-api-access-nr7jt\") pod \"ac3318e5-f88a-40e6-9482-55a0e47c162e\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " Nov 03 11:32:14.673180 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.672921 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-metrics-certs\") pod \"ac3318e5-f88a-40e6-9482-55a0e47c162e\" (UID: \"ac3318e5-f88a-40e6-9482-55a0e47c162e\") " Nov 03 11:32:14.673180 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.673116 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3318e5-f88a-40e6-9482-55a0e47c162e-openshift-service-ca" pod "ac3318e5-f88a-40e6-9482-55a0e47c162e" (UID: "ac3318e5-f88a-40e6-9482-55a0e47c162e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:32:14.673660 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.673500 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3318e5-f88a-40e6-9482-55a0e47c162e-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:14.681750 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.681713 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-metrics-certs" pod "ac3318e5-f88a-40e6-9482-55a0e47c162e" (UID: "ac3318e5-f88a-40e6-9482-55a0e47c162e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:14.681939 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.681800 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3318e5-f88a-40e6-9482-55a0e47c162e-kube-api-access-nr7jt" pod "ac3318e5-f88a-40e6-9482-55a0e47c162e" (UID: "ac3318e5-f88a-40e6-9482-55a0e47c162e"). InnerVolumeSpecName "kube-api-access-nr7jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:32:14.685736 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.685700 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-maas-api-tls" pod "ac3318e5-f88a-40e6-9482-55a0e47c162e" (UID: "ac3318e5-f88a-40e6-9482-55a0e47c162e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:14.788354 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.788240 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:14.788354 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.788277 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr7jt\" (UniqueName: \"kubernetes.io/projected/ac3318e5-f88a-40e6-9482-55a0e47c162e-kube-api-access-nr7jt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:14.788354 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.788309 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3318e5-f88a-40e6-9482-55a0e47c162e-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:14.957438 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.957400 5086 generic.go:356] "Generic (PLEG): container finished" podID="ac3318e5-f88a-40e6-9482-55a0e47c162e" containerID="857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366" exitCode=0 Nov 03 11:32:14.957639 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.957473 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" Nov 03 11:32:14.957639 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.957484 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" event={"ID":"ac3318e5-f88a-40e6-9482-55a0e47c162e","Type":"ContainerDied","Data":"857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366"} Nov 03 11:32:14.957639 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.957534 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r" event={"ID":"ac3318e5-f88a-40e6-9482-55a0e47c162e","Type":"ContainerDied","Data":"74a4e34f428da8a2ff556d8dfd1f97fee823168f0f90e2b9904219b1c391caa7"} Nov 03 11:32:14.957639 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.957564 5086 scope.go:122] "RemoveContainer" containerID="857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366" Nov 03 11:32:14.971651 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.971630 5086 scope.go:122] "RemoveContainer" containerID="857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366" Nov 03 11:32:14.971959 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:32:14.971940 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366\": container with ID starting with 857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366 not found: ID does not exist" containerID="857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366" Nov 03 11:32:14.972020 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.971973 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366"} err="failed to get container status \"857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366\": rpc error: code = NotFound desc = could not find container \"857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366\": container with ID starting with 857246df718138fde27309d64d58bd73c96fb4b9a263000b658aa2a71bd67366 not found: ID does not exist" Nov 03 11:32:14.984060 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.984036 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r"] Nov 03 11:32:14.988978 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:14.988955 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f78621-85d4c55c5c-lcn2r"] Nov 03 11:32:15.809511 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:15.809469 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" podUID="c1228536-409d-41c3-aab5-b70ea3c1f4e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 11:32:16.171213 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:16.171175 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3318e5-f88a-40e6-9482-55a0e47c162e" path="/var/lib/kubelet/pods/ac3318e5-f88a-40e6-9482-55a0e47c162e/volumes" Nov 03 11:32:19.893024 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:19.892991 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn"] Nov 03 11:32:19.893889 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:19.893849 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" podUID="5f032621-ff28-4049-abe9-698a496a82f8" containerName="maas-api" containerID="cri-o://a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12" gracePeriod=30 Nov 03 11:32:20.210451 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.210420 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:32:20.247458 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.247425 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-metrics-certs\") pod \"5f032621-ff28-4049-abe9-698a496a82f8\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " Nov 03 11:32:20.247641 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.247465 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-maas-api-tls\") pod \"5f032621-ff28-4049-abe9-698a496a82f8\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " Nov 03 11:32:20.247641 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.247529 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f032621-ff28-4049-abe9-698a496a82f8-kube-api-access-zqkjq\" (UniqueName: \"kubernetes.io/projected/5f032621-ff28-4049-abe9-698a496a82f8-kube-api-access-zqkjq\") pod \"5f032621-ff28-4049-abe9-698a496a82f8\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " Nov 03 11:32:20.247641 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.247622 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f032621-ff28-4049-abe9-698a496a82f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f032621-ff28-4049-abe9-698a496a82f8-openshift-service-ca\") pod \"5f032621-ff28-4049-abe9-698a496a82f8\" (UID: \"5f032621-ff28-4049-abe9-698a496a82f8\") " Nov 03 11:32:20.248022 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.247992 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f032621-ff28-4049-abe9-698a496a82f8-openshift-service-ca" pod "5f032621-ff28-4049-abe9-698a496a82f8" (UID: "5f032621-ff28-4049-abe9-698a496a82f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:32:20.249568 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.249542 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-metrics-certs" pod "5f032621-ff28-4049-abe9-698a496a82f8" (UID: "5f032621-ff28-4049-abe9-698a496a82f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:20.249830 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.249803 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f032621-ff28-4049-abe9-698a496a82f8-kube-api-access-zqkjq" pod "5f032621-ff28-4049-abe9-698a496a82f8" (UID: "5f032621-ff28-4049-abe9-698a496a82f8"). InnerVolumeSpecName "kube-api-access-zqkjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:32:20.249830 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.249808 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-maas-api-tls" pod "5f032621-ff28-4049-abe9-698a496a82f8" (UID: "5f032621-ff28-4049-abe9-698a496a82f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:20.348815 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.348781 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqkjq\" (UniqueName: \"kubernetes.io/projected/5f032621-ff28-4049-abe9-698a496a82f8-kube-api-access-zqkjq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:20.348815 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.348809 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f032621-ff28-4049-abe9-698a496a82f8-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:20.348815 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.348821 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:20.349051 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.348833 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f032621-ff28-4049-abe9-698a496a82f8-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:20.998789 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.998753 5086 generic.go:356] "Generic (PLEG): container finished" podID="5f032621-ff28-4049-abe9-698a496a82f8" containerID="a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12" exitCode=0 Nov 03 11:32:20.999217 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.998799 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" Nov 03 11:32:20.999217 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.998839 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" event={"ID":"5f032621-ff28-4049-abe9-698a496a82f8","Type":"ContainerDied","Data":"a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12"} Nov 03 11:32:20.999217 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.998892 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" event={"ID":"5f032621-ff28-4049-abe9-698a496a82f8","Type":"ContainerDied","Data":"145e497c3652d0c8a308619fe7b5d15ece4acba43e2b9ab8aac0b48bf55d157b"} Nov 03 11:32:20.999217 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:20.998915 5086 scope.go:122] "RemoveContainer" containerID="a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12" Nov 03 11:32:21.009093 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.009075 5086 scope.go:122] "RemoveContainer" containerID="a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12" Nov 03 11:32:21.009380 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:32:21.009358 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12\": container with ID starting with a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12 not found: ID does not exist" containerID="a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12" Nov 03 11:32:21.009438 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.009389 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12"} err="failed to get container status \"a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12\": rpc error: code = NotFound desc = could not find container \"a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12\": container with ID starting with a0ceafa5ea685838e2b43211673af1add0c4b08cbbda4227d82c7941dff9fd12 not found: ID does not exist" Nov 03 11:32:21.028861 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.028832 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn"] Nov 03 11:32:21.029085 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.028950 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn"] Nov 03 11:32:21.058285 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.058248 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" podUID="40abd8ce-e351-4f5d-8bad-b774e6bd47c3" containerName="main" containerID="cri-o://2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d" gracePeriod=45 Nov 03 11:32:21.365750 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.365724 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:32:21.459527 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.459471 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-model-cache\") pod \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " Nov 03 11:32:21.459740 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.459535 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tls-certs\") pod \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " Nov 03 11:32:21.459740 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.459573 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-kube-api-access-t4wd5\" (UniqueName: \"kubernetes.io/projected/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-kube-api-access-t4wd5\") pod \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " Nov 03 11:32:21.459870 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.459806 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-home\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-home\") pod \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " Nov 03 11:32:21.459926 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.459866 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tmp-dir\") pod \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " Nov 03 11:32:21.459926 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.459807 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-model-cache" pod "40abd8ce-e351-4f5d-8bad-b774e6bd47c3" (UID: "40abd8ce-e351-4f5d-8bad-b774e6bd47c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:32:21.459926 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.459896 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-dshm\") pod \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\" (UID: \"40abd8ce-e351-4f5d-8bad-b774e6bd47c3\") " Nov 03 11:32:21.460142 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.460109 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tmp-dir" pod "40abd8ce-e351-4f5d-8bad-b774e6bd47c3" (UID: "40abd8ce-e351-4f5d-8bad-b774e6bd47c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:32:21.460284 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.460182 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-home" pod "40abd8ce-e351-4f5d-8bad-b774e6bd47c3" (UID: "40abd8ce-e351-4f5d-8bad-b774e6bd47c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:32:21.460523 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.460500 5086 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:21.460523 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.460525 5086 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:21.460758 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.460539 5086 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:21.462580 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.462552 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tls-certs" pod "40abd8ce-e351-4f5d-8bad-b774e6bd47c3" (UID: "40abd8ce-e351-4f5d-8bad-b774e6bd47c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:21.462747 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.462726 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-dshm" pod "40abd8ce-e351-4f5d-8bad-b774e6bd47c3" (UID: "40abd8ce-e351-4f5d-8bad-b774e6bd47c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:32:21.462817 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.462735 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-kube-api-access-t4wd5" pod "40abd8ce-e351-4f5d-8bad-b774e6bd47c3" (UID: "40abd8ce-e351-4f5d-8bad-b774e6bd47c3"). InnerVolumeSpecName "kube-api-access-t4wd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:32:21.561218 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.561137 5086 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:21.561218 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.561164 5086 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:21.561218 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:21.561174 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wd5\" (UniqueName: \"kubernetes.io/projected/40abd8ce-e351-4f5d-8bad-b774e6bd47c3-kube-api-access-t4wd5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:22.006959 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.006719 5086 generic.go:356] "Generic (PLEG): container finished" podID="40abd8ce-e351-4f5d-8bad-b774e6bd47c3" containerID="2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d" exitCode=0 Nov 03 11:32:22.006959 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.006789 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" Nov 03 11:32:22.006959 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.006819 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" event={"ID":"40abd8ce-e351-4f5d-8bad-b774e6bd47c3","Type":"ContainerDied","Data":"2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d"} Nov 03 11:32:22.006959 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.006850 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v" event={"ID":"40abd8ce-e351-4f5d-8bad-b774e6bd47c3","Type":"ContainerDied","Data":"5001aaf5f431c3eb3c203794d5fbbeb57485a05d110b101b5293f2da7ab37736"} Nov 03 11:32:22.006959 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.006867 5086 scope.go:122] "RemoveContainer" containerID="2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d" Nov 03 11:32:22.018077 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.018057 5086 scope.go:122] "RemoveContainer" containerID="2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d" Nov 03 11:32:22.018397 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:32:22.018367 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d\": container with ID starting with 2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d not found: ID does not exist" containerID="2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d" Nov 03 11:32:22.018511 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.018406 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d"} err="failed to get container status \"2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d\": rpc error: code = NotFound desc = could not find container \"2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d\": container with ID starting with 2cb399f11c2d59ca40cd7cef0e6374f5bf03c10c614a8c91b9fa6c082c3b333d not found: ID does not exist" Nov 03 11:32:22.028805 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.028785 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v"] Nov 03 11:32:22.031979 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.031957 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/sub-test-model-ce5555-kserve-77696994fc-wxd6v"] Nov 03 11:32:22.172406 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.172371 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40abd8ce-e351-4f5d-8bad-b774e6bd47c3" path="/var/lib/kubelet/pods/40abd8ce-e351-4f5d-8bad-b774e6bd47c3/volumes" Nov 03 11:32:22.172800 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:22.172786 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f032621-ff28-4049-abe9-698a496a82f8" path="/var/lib/kubelet/pods/5f032621-ff28-4049-abe9-698a496a82f8/volumes" Nov 03 11:32:23.109047 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:23.108964 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-35936b-6f6987f468-75nrn" podUID="5f032621-ff28-4049-abe9-698a496a82f8" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.187:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 11:32:24.293876 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:32:24.293838 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:32:25.810342 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:25.810278 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" podUID="c1228536-409d-41c3-aab5-b70ea3c1f4e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 11:32:31.099341 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.099277 5086 generic.go:356] "Generic (PLEG): container finished" podID="c1228536-409d-41c3-aab5-b70ea3c1f4e6" containerID="48b0f73f069fc392f03a44cb89ff41b45a99a274c407882c38df58ef849dfe82" exitCode=0 Nov 03 11:32:31.099680 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.099336 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" event={"ID":"c1228536-409d-41c3-aab5-b70ea3c1f4e6","Type":"ContainerDied","Data":"48b0f73f069fc392f03a44cb89ff41b45a99a274c407882c38df58ef849dfe82"} Nov 03 11:32:31.147411 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.147006 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 11:32:31.284567 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.284537 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1228536-409d-41c3-aab5-b70ea3c1f4e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1228536-409d-41c3-aab5-b70ea3c1f4e6-plugins-config-volume\") pod \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\" (UID: \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\") " Nov 03 11:32:31.284749 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.284709 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1228536-409d-41c3-aab5-b70ea3c1f4e6-kube-api-access-zqxcz\" (UniqueName: \"kubernetes.io/projected/c1228536-409d-41c3-aab5-b70ea3c1f4e6-kube-api-access-zqxcz\") pod \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\" (UID: \"c1228536-409d-41c3-aab5-b70ea3c1f4e6\") " Nov 03 11:32:31.284946 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.284922 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1228536-409d-41c3-aab5-b70ea3c1f4e6-plugins-config-volume" pod "c1228536-409d-41c3-aab5-b70ea3c1f4e6" (UID: "c1228536-409d-41c3-aab5-b70ea3c1f4e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:32:31.285054 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.285038 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1228536-409d-41c3-aab5-b70ea3c1f4e6-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:31.286935 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.286915 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1228536-409d-41c3-aab5-b70ea3c1f4e6-kube-api-access-zqxcz" pod "c1228536-409d-41c3-aab5-b70ea3c1f4e6" (UID: "c1228536-409d-41c3-aab5-b70ea3c1f4e6"). InnerVolumeSpecName "kube-api-access-zqxcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:32:31.385731 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.385661 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqxcz\" (UniqueName: \"kubernetes.io/projected/c1228536-409d-41c3-aab5-b70ea3c1f4e6-kube-api-access-zqxcz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:31.678190 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.678096 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx"] Nov 03 11:32:31.678407 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.678371 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" podUID="b70193f5-6efd-4009-a8a3-2d1405f4ce7c" containerName="maas-api" containerID="cri-o://41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550" gracePeriod=30 Nov 03 11:32:31.969529 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:31.969506 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 11:32:32.094709 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.094670 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-maas-api-tls\") pod \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " Nov 03 11:32:32.094911 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.094747 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-metrics-certs\") pod \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " Nov 03 11:32:32.094911 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.094854 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-openshift-service-ca\") pod \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " Nov 03 11:32:32.095043 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.094914 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-kube-api-access-76t5b\" (UniqueName: \"kubernetes.io/projected/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-kube-api-access-76t5b\") pod \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\" (UID: \"b70193f5-6efd-4009-a8a3-2d1405f4ce7c\") " Nov 03 11:32:32.095363 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.095279 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-openshift-service-ca" pod "b70193f5-6efd-4009-a8a3-2d1405f4ce7c" (UID: "b70193f5-6efd-4009-a8a3-2d1405f4ce7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:32:32.096914 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.096888 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-maas-api-tls" pod "b70193f5-6efd-4009-a8a3-2d1405f4ce7c" (UID: "b70193f5-6efd-4009-a8a3-2d1405f4ce7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:32.097027 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.096986 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-metrics-certs" pod "b70193f5-6efd-4009-a8a3-2d1405f4ce7c" (UID: "b70193f5-6efd-4009-a8a3-2d1405f4ce7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:32.097192 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.097176 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-kube-api-access-76t5b" pod "b70193f5-6efd-4009-a8a3-2d1405f4ce7c" (UID: "b70193f5-6efd-4009-a8a3-2d1405f4ce7c"). InnerVolumeSpecName "kube-api-access-76t5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:32:32.107977 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.107948 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" Nov 03 11:32:32.107977 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.107956 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b" event={"ID":"c1228536-409d-41c3-aab5-b70ea3c1f4e6","Type":"ContainerDied","Data":"ae659cf9b3707afaf249a472edff8339caa7dff0657b3aeaf7f9e198e4e0cdf2"} Nov 03 11:32:32.108423 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.108025 5086 scope.go:122] "RemoveContainer" containerID="48b0f73f069fc392f03a44cb89ff41b45a99a274c407882c38df58ef849dfe82" Nov 03 11:32:32.112572 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.112542 5086 generic.go:356] "Generic (PLEG): container finished" podID="b70193f5-6efd-4009-a8a3-2d1405f4ce7c" containerID="41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550" exitCode=0 Nov 03 11:32:32.112659 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.112635 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" Nov 03 11:32:32.112659 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.112644 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" event={"ID":"b70193f5-6efd-4009-a8a3-2d1405f4ce7c","Type":"ContainerDied","Data":"41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550"} Nov 03 11:32:32.112749 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.112688 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx" event={"ID":"b70193f5-6efd-4009-a8a3-2d1405f4ce7c","Type":"ContainerDied","Data":"397eab29292a6ea639d68c74fa283126497871399e41d5f63e08941d7f3e02ce"} Nov 03 11:32:32.122822 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.122804 5086 scope.go:122] "RemoveContainer" containerID="41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550" Nov 03 11:32:32.132461 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.132435 5086 scope.go:122] "RemoveContainer" containerID="41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550" Nov 03 11:32:32.132832 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:32:32.132810 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550\": container with ID starting with 41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550 not found: ID does not exist" containerID="41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550" Nov 03 11:32:32.132906 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.132842 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550"} err="failed to get container status \"41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550\": rpc error: code = NotFound desc = could not find container \"41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550\": container with ID starting with 41aca87579cfb809e3d8d4089866649ac04515294bf6d0a5356b8335f6253550 not found: ID does not exist" Nov 03 11:32:32.134242 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.134221 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b"] Nov 03 11:32:32.137914 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.137895 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6012b5-6887bc75f5-wwj4b"] Nov 03 11:32:32.160931 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.160878 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx"] Nov 03 11:32:32.171231 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.171201 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1228536-409d-41c3-aab5-b70ea3c1f4e6" path="/var/lib/kubelet/pods/c1228536-409d-41c3-aab5-b70ea3c1f4e6/volumes" Nov 03 11:32:32.171541 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.171524 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cc8214-55fcd965f7-p5rpx"] Nov 03 11:32:32.196917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.196853 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:32.196917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.196876 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:32.196917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.196887 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:32.196917 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:32.196896 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76t5b\" (UniqueName: \"kubernetes.io/projected/b70193f5-6efd-4009-a8a3-2d1405f4ce7c-kube-api-access-76t5b\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:34.170469 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:34.170436 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70193f5-6efd-4009-a8a3-2d1405f4ce7c" path="/var/lib/kubelet/pods/b70193f5-6efd-4009-a8a3-2d1405f4ce7c/volumes" Nov 03 11:32:42.634011 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:42.633973 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v"] Nov 03 11:32:42.634659 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:42.634205 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" podUID="0602f904-1cbe-4088-9832-0961e0088df9" containerName="maas-api" containerID="cri-o://c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a" gracePeriod=30 Nov 03 11:32:42.703350 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:42.703285 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" podUID="0602f904-1cbe-4088-9832-0961e0088df9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.188:8443/health\": dial tcp 10.132.0.188:8443: connect: connection refused" Nov 03 11:32:42.985115 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:42.985087 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:32:43.129449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.125122 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0602f904-1cbe-4088-9832-0961e0088df9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0602f904-1cbe-4088-9832-0961e0088df9-openshift-service-ca\") pod \"0602f904-1cbe-4088-9832-0961e0088df9\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " Nov 03 11:32:43.129449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.125229 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0602f904-1cbe-4088-9832-0961e0088df9-kube-api-access-q86pn\" (UniqueName: \"kubernetes.io/projected/0602f904-1cbe-4088-9832-0961e0088df9-kube-api-access-q86pn\") pod \"0602f904-1cbe-4088-9832-0961e0088df9\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " Nov 03 11:32:43.129449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.125384 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls\") pod \"0602f904-1cbe-4088-9832-0961e0088df9\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " Nov 03 11:32:43.129449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.125438 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-metrics-certs\") pod \"0602f904-1cbe-4088-9832-0961e0088df9\" (UID: \"0602f904-1cbe-4088-9832-0961e0088df9\") " Nov 03 11:32:43.129449 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.127595 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0602f904-1cbe-4088-9832-0961e0088df9-openshift-service-ca" pod "0602f904-1cbe-4088-9832-0961e0088df9" (UID: "0602f904-1cbe-4088-9832-0961e0088df9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:32:43.133566 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.132327 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-metrics-certs" pod "0602f904-1cbe-4088-9832-0961e0088df9" (UID: "0602f904-1cbe-4088-9832-0961e0088df9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:43.133566 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.132332 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0602f904-1cbe-4088-9832-0961e0088df9-kube-api-access-q86pn" pod "0602f904-1cbe-4088-9832-0961e0088df9" (UID: "0602f904-1cbe-4088-9832-0961e0088df9"). InnerVolumeSpecName "kube-api-access-q86pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:32:43.134388 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.134359 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls" pod "0602f904-1cbe-4088-9832-0961e0088df9" (UID: "0602f904-1cbe-4088-9832-0961e0088df9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:32:43.199701 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.199667 5086 generic.go:356] "Generic (PLEG): container finished" podID="0602f904-1cbe-4088-9832-0961e0088df9" containerID="c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a" exitCode=0 Nov 03 11:32:43.199862 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.199720 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" Nov 03 11:32:43.199862 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.199729 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" event={"ID":"0602f904-1cbe-4088-9832-0961e0088df9","Type":"ContainerDied","Data":"c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a"} Nov 03 11:32:43.199862 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.199779 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v" event={"ID":"0602f904-1cbe-4088-9832-0961e0088df9","Type":"ContainerDied","Data":"67d8c392a909498d33ad18769e0b4f7e15ec2849a0f425b51825c4ca2d2060f6"} Nov 03 11:32:43.199862 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.199810 5086 scope.go:122] "RemoveContainer" containerID="c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a" Nov 03 11:32:43.210663 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.210644 5086 scope.go:122] "RemoveContainer" containerID="c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a" Nov 03 11:32:43.210941 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:32:43.210923 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a\": container with ID starting with c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a not found: ID does not exist" containerID="c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a" Nov 03 11:32:43.210990 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.210949 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a"} err="failed to get container status \"c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a\": rpc error: code = NotFound desc = could not find container \"c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a\": container with ID starting with c6e0c8e5109e89b1ff61c79246440b2a5543bf318960d00807543b761940b15a not found: ID does not exist" Nov 03 11:32:43.226033 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.226010 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v"] Nov 03 11:32:43.227750 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.227721 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:43.227831 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.227756 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0602f904-1cbe-4088-9832-0961e0088df9-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:43.227831 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.227775 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0602f904-1cbe-4088-9832-0961e0088df9-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:43.227831 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.227790 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q86pn\" (UniqueName: \"kubernetes.io/projected/0602f904-1cbe-4088-9832-0961e0088df9-kube-api-access-q86pn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:32:43.231390 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:43.231365 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5711cf-57f98cc67b-5jr8v"] Nov 03 11:32:44.170404 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:32:44.170365 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0602f904-1cbe-4088-9832-0961e0088df9" path="/var/lib/kubelet/pods/0602f904-1cbe-4088-9832-0961e0088df9/volumes" Nov 03 11:32:54.275681 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:32:54.275640 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:33:00.225690 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:00.225658 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc"] Nov 03 11:33:00.243732 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:00.243697 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7"] Nov 03 11:33:03.100872 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.100836 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r"] Nov 03 11:33:03.101372 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.101062 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" podUID="20fc1bd1-f950-46a7-8ec3-54555a42e1d0" containerName="maas-api" containerID="cri-o://2dccc69098e8ccf41d1a80bf143dc803c45685bcf9599509576e0c27e5caef73" gracePeriod=30 Nov 03 11:33:03.334193 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.334156 5086 generic.go:356] "Generic (PLEG): container finished" podID="20fc1bd1-f950-46a7-8ec3-54555a42e1d0" containerID="2dccc69098e8ccf41d1a80bf143dc803c45685bcf9599509576e0c27e5caef73" exitCode=0 Nov 03 11:33:03.334445 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.334362 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" event={"ID":"20fc1bd1-f950-46a7-8ec3-54555a42e1d0","Type":"ContainerDied","Data":"2dccc69098e8ccf41d1a80bf143dc803c45685bcf9599509576e0c27e5caef73"} Nov 03 11:33:03.390350 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.390290 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 11:33:03.444628 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.444597 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-maas-api-tls\") pod \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " Nov 03 11:33:03.444798 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.444758 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-kube-api-access-j86mb\" (UniqueName: \"kubernetes.io/projected/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-kube-api-access-j86mb\") pod \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " Nov 03 11:33:03.444844 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.444801 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-openshift-service-ca\") pod \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " Nov 03 11:33:03.444890 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.444839 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-metrics-certs\") pod \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\" (UID: \"20fc1bd1-f950-46a7-8ec3-54555a42e1d0\") " Nov 03 11:33:03.445254 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.445220 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-openshift-service-ca" pod "20fc1bd1-f950-46a7-8ec3-54555a42e1d0" (UID: "20fc1bd1-f950-46a7-8ec3-54555a42e1d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:33:03.446859 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.446830 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-maas-api-tls" pod "20fc1bd1-f950-46a7-8ec3-54555a42e1d0" (UID: "20fc1bd1-f950-46a7-8ec3-54555a42e1d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:33:03.447444 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.447419 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-metrics-certs" pod "20fc1bd1-f950-46a7-8ec3-54555a42e1d0" (UID: "20fc1bd1-f950-46a7-8ec3-54555a42e1d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:33:03.447554 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.447451 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-kube-api-access-j86mb" pod "20fc1bd1-f950-46a7-8ec3-54555a42e1d0" (UID: "20fc1bd1-f950-46a7-8ec3-54555a42e1d0"). InnerVolumeSpecName "kube-api-access-j86mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:33:03.546140 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.546107 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86mb\" (UniqueName: \"kubernetes.io/projected/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-kube-api-access-j86mb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:03.546140 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.546135 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:03.546140 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.546146 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:03.546425 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:03.546156 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc1bd1-f950-46a7-8ec3-54555a42e1d0-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:04.341351 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:04.341291 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" Nov 03 11:33:04.341842 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:04.341286 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r" event={"ID":"20fc1bd1-f950-46a7-8ec3-54555a42e1d0","Type":"ContainerDied","Data":"0179954c429f76d17f687ba49a5f9f2fc7c7e7f2c3d7f98cbce8ffdd488dadbd"} Nov 03 11:33:04.341842 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:04.341449 5086 scope.go:122] "RemoveContainer" containerID="2dccc69098e8ccf41d1a80bf143dc803c45685bcf9599509576e0c27e5caef73" Nov 03 11:33:04.360437 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:04.360408 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r"] Nov 03 11:33:04.366582 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:04.366551 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ce5555-5cfb4c77dc-pkw8r"] Nov 03 11:33:05.227879 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.227838 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" podUID="6e657caa-0b2b-4318-ac6f-1d1e811992d5" containerName="payload-processing" containerID="cri-o://b63b714cc03530e21f7322f473c9dc353f74f67110bd862a61f068d13049c148" gracePeriod=25 Nov 03 11:33:05.244507 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.244479 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" podUID="52ef7069-f15c-4d0e-89d3-2724ff2a0fb2" containerName="payload-pre-processing" containerID="cri-o://b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de" gracePeriod=25 Nov 03 11:33:05.350033 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.349651 5086 generic.go:356] "Generic (PLEG): container finished" podID="6e657caa-0b2b-4318-ac6f-1d1e811992d5" containerID="b63b714cc03530e21f7322f473c9dc353f74f67110bd862a61f068d13049c148" exitCode=0 Nov 03 11:33:05.350033 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.349715 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" event={"ID":"6e657caa-0b2b-4318-ac6f-1d1e811992d5","Type":"ContainerDied","Data":"b63b714cc03530e21f7322f473c9dc353f74f67110bd862a61f068d13049c148"} Nov 03 11:33:05.510815 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.510789 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 11:33:05.541316 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.541281 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 11:33:05.568776 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.568746 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e657caa-0b2b-4318-ac6f-1d1e811992d5-kube-api-access-8v7jx\" (UniqueName: \"kubernetes.io/projected/6e657caa-0b2b-4318-ac6f-1d1e811992d5-kube-api-access-8v7jx\") pod \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\" (UID: \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\") " Nov 03 11:33:05.568953 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.568856 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e657caa-0b2b-4318-ac6f-1d1e811992d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e657caa-0b2b-4318-ac6f-1d1e811992d5-plugins-config-volume\") pod \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\" (UID: \"6e657caa-0b2b-4318-ac6f-1d1e811992d5\") " Nov 03 11:33:05.569281 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.569259 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e657caa-0b2b-4318-ac6f-1d1e811992d5-plugins-config-volume" pod "6e657caa-0b2b-4318-ac6f-1d1e811992d5" (UID: "6e657caa-0b2b-4318-ac6f-1d1e811992d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:33:05.571464 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.571434 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e657caa-0b2b-4318-ac6f-1d1e811992d5-kube-api-access-8v7jx" pod "6e657caa-0b2b-4318-ac6f-1d1e811992d5" (UID: "6e657caa-0b2b-4318-ac6f-1d1e811992d5"). InnerVolumeSpecName "kube-api-access-8v7jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:33:05.669950 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.669908 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-plugins-config-volume\") pod \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\" (UID: \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\") " Nov 03 11:33:05.670137 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.670014 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-kube-api-access-gq46c\" (UniqueName: \"kubernetes.io/projected/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-kube-api-access-gq46c\") pod \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\" (UID: \"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2\") " Nov 03 11:33:05.670457 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.670433 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v7jx\" (UniqueName: \"kubernetes.io/projected/6e657caa-0b2b-4318-ac6f-1d1e811992d5-kube-api-access-8v7jx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:05.670583 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.670463 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e657caa-0b2b-4318-ac6f-1d1e811992d5-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:05.670583 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.670430 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-plugins-config-volume" pod "52ef7069-f15c-4d0e-89d3-2724ff2a0fb2" (UID: "52ef7069-f15c-4d0e-89d3-2724ff2a0fb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:33:05.672209 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.672186 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-kube-api-access-gq46c" pod "52ef7069-f15c-4d0e-89d3-2724ff2a0fb2" (UID: "52ef7069-f15c-4d0e-89d3-2724ff2a0fb2"). InnerVolumeSpecName "kube-api-access-gq46c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:33:05.771082 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.770998 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq46c\" (UniqueName: \"kubernetes.io/projected/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-kube-api-access-gq46c\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:05.771082 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:05.771030 5086 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:06.171106 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.171074 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20fc1bd1-f950-46a7-8ec3-54555a42e1d0" path="/var/lib/kubelet/pods/20fc1bd1-f950-46a7-8ec3-54555a42e1d0/volumes" Nov 03 11:33:06.356450 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.356417 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" Nov 03 11:33:06.356450 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.356436 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc" event={"ID":"6e657caa-0b2b-4318-ac6f-1d1e811992d5","Type":"ContainerDied","Data":"24b61e0ca1cd5dcd43f20c9ad338af4c80c16041b4da1934d4dbd9a1151365ce"} Nov 03 11:33:06.356928 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.356503 5086 scope.go:122] "RemoveContainer" containerID="b63b714cc03530e21f7322f473c9dc353f74f67110bd862a61f068d13049c148" Nov 03 11:33:06.358189 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.358168 5086 generic.go:356] "Generic (PLEG): container finished" podID="52ef7069-f15c-4d0e-89d3-2724ff2a0fb2" containerID="b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de" exitCode=0 Nov 03 11:33:06.358320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.358198 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" event={"ID":"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2","Type":"ContainerDied","Data":"b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de"} Nov 03 11:33:06.358320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.358234 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" Nov 03 11:33:06.358320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.358245 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7" event={"ID":"52ef7069-f15c-4d0e-89d3-2724ff2a0fb2","Type":"ContainerDied","Data":"eeb09d15304a0875b5f1848c7e59754a55565fd26912516440ecfcbe9f080756"} Nov 03 11:33:06.366136 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.366121 5086 scope.go:122] "RemoveContainer" containerID="b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de" Nov 03 11:33:06.374568 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.374548 5086 scope.go:122] "RemoveContainer" containerID="b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de" Nov 03 11:33:06.374874 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:33:06.374827 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de\": container with ID starting with b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de not found: ID does not exist" containerID="b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de" Nov 03 11:33:06.374874 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.374864 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de"} err="failed to get container status \"b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de\": rpc error: code = NotFound desc = could not find container \"b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de\": container with ID starting with b65fbe6191efdeb819c2009812f7daa0657be00fddfbada2b02aed9b0d05a1de not found: ID does not exist" Nov 03 11:33:06.377621 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.375524 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7"] Nov 03 11:33:06.377953 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.377937 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010232-dd4979d95-h6pg7"] Nov 03 11:33:06.388878 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.388855 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc"] Nov 03 11:33:06.390746 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:06.390727 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010232-6bf5dbf45c-zn8sc"] Nov 03 11:33:08.170785 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:08.170748 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ef7069-f15c-4d0e-89d3-2724ff2a0fb2" path="/var/lib/kubelet/pods/52ef7069-f15c-4d0e-89d3-2724ff2a0fb2/volumes" Nov 03 11:33:08.171400 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:08.171378 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e657caa-0b2b-4318-ac6f-1d1e811992d5" path="/var/lib/kubelet/pods/6e657caa-0b2b-4318-ac6f-1d1e811992d5/volumes" Nov 03 11:33:20.179595 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.179200 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5"] Nov 03 11:33:20.179595 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.179569 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" podUID="1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" containerName="maas-api" containerID="cri-o://813df2b8fe2109da4b91e3fe85e6eae71ec75bfc7884a16af7566475d70fde9d" gracePeriod=30 Nov 03 11:33:20.485392 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.484128 5086 generic.go:356] "Generic (PLEG): container finished" podID="1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" containerID="813df2b8fe2109da4b91e3fe85e6eae71ec75bfc7884a16af7566475d70fde9d" exitCode=0 Nov 03 11:33:20.485392 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.484221 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" event={"ID":"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d","Type":"ContainerDied","Data":"813df2b8fe2109da4b91e3fe85e6eae71ec75bfc7884a16af7566475d70fde9d"} Nov 03 11:33:20.485392 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.484281 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" event={"ID":"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d","Type":"ContainerDied","Data":"0d1cd00d88407d905667d89d637a14a477c7922f46fb1aa4ebd54c49cee4f973"} Nov 03 11:33:20.485392 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.484505 5086 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d1cd00d88407d905667d89d637a14a477c7922f46fb1aa4ebd54c49cee4f973" Nov 03 11:33:20.485392 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.484847 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 11:33:20.628483 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.628446 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-kube-api-access-cs6fq\" (UniqueName: \"kubernetes.io/projected/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-kube-api-access-cs6fq\") pod \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " Nov 03 11:33:20.628677 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.628518 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-openshift-service-ca\") pod \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " Nov 03 11:33:20.628677 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.628561 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-maas-api-tls\") pod \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " Nov 03 11:33:20.628677 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.628604 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-metrics-certs\") pod \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\" (UID: \"1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d\") " Nov 03 11:33:20.628887 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.628864 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-openshift-service-ca" pod "1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" (UID: "1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:33:20.629016 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.628998 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:20.630732 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.630699 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-maas-api-tls" pod "1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" (UID: "1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:33:20.630838 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.630770 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-metrics-certs" pod "1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" (UID: "1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:33:20.630838 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.630784 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-kube-api-access-cs6fq" pod "1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" (UID: "1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d"). InnerVolumeSpecName "kube-api-access-cs6fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:33:20.730278 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.730245 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs6fq\" (UniqueName: \"kubernetes.io/projected/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-kube-api-access-cs6fq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:20.730278 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.730276 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:20.730278 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:20.730286 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:21.490147 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:21.490110 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5" Nov 03 11:33:21.511978 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:21.511947 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5"] Nov 03 11:33:21.517603 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:21.517576 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-010232-68868b5b56-mgbp5"] Nov 03 11:33:22.169491 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:22.169455 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" path="/var/lib/kubelet/pods/1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d/volumes" Nov 03 11:33:24.273084 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:33:24.273049 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:33:27.736951 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:27.736917 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t"] Nov 03 11:33:27.737790 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:27.737759 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" containerID="cri-o://63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375" gracePeriod=30 Nov 03 11:33:32.999029 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:32.999006 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 11:33:33.157500 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157471 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-certs\") pod \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " Nov 03 11:33:33.157704 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157565 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-kube-api-access-wh52z\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-kube-api-access-wh52z\") pod \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " Nov 03 11:33:33.157704 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157603 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-credential-socket\") pod \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " Nov 03 11:33:33.157704 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157647 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istiod-ca-cert\") pod \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " Nov 03 11:33:33.157897 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157737 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-podinfo\") pod \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " Nov 03 11:33:33.157897 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157761 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-certs" pod "f96bd9cc-e47f-40b3-ad46-2bd17d497592" (UID: "f96bd9cc-e47f-40b3-ad46-2bd17d497592"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:33:33.157897 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157787 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-token\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-token\") pod \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " Nov 03 11:33:33.158078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157918 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-socket\") pod \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " Nov 03 11:33:33.158078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157930 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-credential-socket" pod "f96bd9cc-e47f-40b3-ad46-2bd17d497592" (UID: "f96bd9cc-e47f-40b3-ad46-2bd17d497592"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:33:33.158078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.157978 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-data\") pod \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " Nov 03 11:33:33.158078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.158013 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-envoy\") pod \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\" (UID: \"f96bd9cc-e47f-40b3-ad46-2bd17d497592\") " Nov 03 11:33:33.158078 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.158045 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istiod-ca-cert" pod "f96bd9cc-e47f-40b3-ad46-2bd17d497592" (UID: "f96bd9cc-e47f-40b3-ad46-2bd17d497592"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:33:33.158429 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.158196 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-socket" pod "f96bd9cc-e47f-40b3-ad46-2bd17d497592" (UID: "f96bd9cc-e47f-40b3-ad46-2bd17d497592"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:33:33.158429 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.158204 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-data" pod "f96bd9cc-e47f-40b3-ad46-2bd17d497592" (UID: "f96bd9cc-e47f-40b3-ad46-2bd17d497592"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:33:33.158605 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.158487 5086 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.158605 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.158510 5086 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.158605 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.158526 5086 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.158605 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.158540 5086 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.158605 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.158558 5086 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.159902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.159872 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-kube-api-access-wh52z" pod "f96bd9cc-e47f-40b3-ad46-2bd17d497592" (UID: "f96bd9cc-e47f-40b3-ad46-2bd17d497592"). InnerVolumeSpecName "kube-api-access-wh52z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:33:33.159902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.159886 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-podinfo" pod "f96bd9cc-e47f-40b3-ad46-2bd17d497592" (UID: "f96bd9cc-e47f-40b3-ad46-2bd17d497592"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 11:33:33.160050 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.159943 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-token" pod "f96bd9cc-e47f-40b3-ad46-2bd17d497592" (UID: "f96bd9cc-e47f-40b3-ad46-2bd17d497592"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:33:33.160205 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.160191 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-envoy" pod "f96bd9cc-e47f-40b3-ad46-2bd17d497592" (UID: "f96bd9cc-e47f-40b3-ad46-2bd17d497592"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:33:33.259770 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.259681 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh52z\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-kube-api-access-wh52z\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.259770 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.259725 5086 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.259770 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.259740 5086 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.259770 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.259753 5086 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bd9cc-e47f-40b3-ad46-2bd17d497592-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.277203 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.277166 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n"] Nov 03 11:33:33.277494 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.277445 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" podUID="ea69424b-3f91-47a6-b534-604ca3d09332" containerName="maas-api" containerID="cri-o://6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f" gracePeriod=30 Nov 03 11:33:33.528502 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.528474 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 11:33:33.572185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.571229 5086 generic.go:356] "Generic (PLEG): container finished" podID="ea69424b-3f91-47a6-b534-604ca3d09332" containerID="6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f" exitCode=0 Nov 03 11:33:33.572185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.571356 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" event={"ID":"ea69424b-3f91-47a6-b534-604ca3d09332","Type":"ContainerDied","Data":"6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f"} Nov 03 11:33:33.572185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.571407 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" event={"ID":"ea69424b-3f91-47a6-b534-604ca3d09332","Type":"ContainerDied","Data":"73398049ef1ebdae6fa5c4074e4c2617fcf1afda9acb3bb00767ab2c4ee19f3f"} Nov 03 11:33:33.572185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.571438 5086 scope.go:122] "RemoveContainer" containerID="6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f" Nov 03 11:33:33.572185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.571580 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n" Nov 03 11:33:33.580415 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.576667 5086 generic.go:356] "Generic (PLEG): container finished" podID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerID="63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375" exitCode=0 Nov 03 11:33:33.580415 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.576785 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" event={"ID":"f96bd9cc-e47f-40b3-ad46-2bd17d497592","Type":"ContainerDied","Data":"63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375"} Nov 03 11:33:33.580415 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.576828 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" event={"ID":"f96bd9cc-e47f-40b3-ad46-2bd17d497592","Type":"ContainerDied","Data":"11a58d907579b9dc3d134ffa521b06cf51659d4d49b6d55695c5fe0c76be6a1b"} Nov 03 11:33:33.580415 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.576900 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t" Nov 03 11:33:33.599462 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.599350 5086 scope.go:122] "RemoveContainer" containerID="6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f" Nov 03 11:33:33.600243 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:33:33.600077 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f\": container with ID starting with 6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f not found: ID does not exist" containerID="6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f" Nov 03 11:33:33.600243 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.600128 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f"} err="failed to get container status \"6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f\": rpc error: code = NotFound desc = could not find container \"6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f\": container with ID starting with 6a72bf5e0f9501e747932d3c3c86b8c9a63d0b8e2dfdd5899e6b20c34df21d0f not found: ID does not exist" Nov 03 11:33:33.600243 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.600154 5086 scope.go:122] "RemoveContainer" containerID="63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375" Nov 03 11:33:33.613284 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.613262 5086 scope.go:122] "RemoveContainer" containerID="63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375" Nov 03 11:33:33.613707 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:33:33.613677 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375\": container with ID starting with 63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375 not found: ID does not exist" containerID="63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375" Nov 03 11:33:33.613825 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.613718 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375"} err="failed to get container status \"63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375\": rpc error: code = NotFound desc = could not find container \"63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375\": container with ID starting with 63e5bd2a4ba033cb27aca9ca50d4d3fba4ee7a8729f8b668a4b1302d84680375 not found: ID does not exist" Nov 03 11:33:33.622489 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.622464 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t"] Nov 03 11:33:33.627175 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.627151 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8hn6t"] Nov 03 11:33:33.665650 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.665421 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-metrics-certs\") pod \"ea69424b-3f91-47a6-b534-604ca3d09332\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " Nov 03 11:33:33.665650 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.665539 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-maas-api-tls\") pod \"ea69424b-3f91-47a6-b534-604ca3d09332\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " Nov 03 11:33:33.665650 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.665598 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea69424b-3f91-47a6-b534-604ca3d09332-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69424b-3f91-47a6-b534-604ca3d09332-openshift-service-ca\") pod \"ea69424b-3f91-47a6-b534-604ca3d09332\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " Nov 03 11:33:33.665951 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.665660 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea69424b-3f91-47a6-b534-604ca3d09332-kube-api-access-lzp7x\" (UniqueName: \"kubernetes.io/projected/ea69424b-3f91-47a6-b534-604ca3d09332-kube-api-access-lzp7x\") pod \"ea69424b-3f91-47a6-b534-604ca3d09332\" (UID: \"ea69424b-3f91-47a6-b534-604ca3d09332\") " Nov 03 11:33:33.666363 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.666335 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea69424b-3f91-47a6-b534-604ca3d09332-openshift-service-ca" pod "ea69424b-3f91-47a6-b534-604ca3d09332" (UID: "ea69424b-3f91-47a6-b534-604ca3d09332"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:33:33.668165 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.668135 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-maas-api-tls" pod "ea69424b-3f91-47a6-b534-604ca3d09332" (UID: "ea69424b-3f91-47a6-b534-604ca3d09332"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:33:33.668265 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.668169 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-metrics-certs" pod "ea69424b-3f91-47a6-b534-604ca3d09332" (UID: "ea69424b-3f91-47a6-b534-604ca3d09332"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:33:33.668265 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.668210 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea69424b-3f91-47a6-b534-604ca3d09332-kube-api-access-lzp7x" pod "ea69424b-3f91-47a6-b534-604ca3d09332" (UID: "ea69424b-3f91-47a6-b534-604ca3d09332"). InnerVolumeSpecName "kube-api-access-lzp7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:33:33.766970 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.766899 5086 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.766970 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.766927 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69424b-3f91-47a6-b534-604ca3d09332-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.766970 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.766939 5086 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69424b-3f91-47a6-b534-604ca3d09332-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.766970 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.766949 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzp7x\" (UniqueName: \"kubernetes.io/projected/ea69424b-3f91-47a6-b534-604ca3d09332-kube-api-access-lzp7x\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:33.898381 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.898145 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n"] Nov 03 11:33:33.900554 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:33.900517 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-836589-bc656f8b9-6cg7n"] Nov 03 11:33:34.171480 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:34.171448 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea69424b-3f91-47a6-b534-604ca3d09332" path="/var/lib/kubelet/pods/ea69424b-3f91-47a6-b534-604ca3d09332/volumes" Nov 03 11:33:34.171936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:34.171921 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" path="/var/lib/kubelet/pods/f96bd9cc-e47f-40b3-ad46-2bd17d497592/volumes" Nov 03 11:33:42.056148 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056107 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt"] Nov 03 11:33:42.056592 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056573 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d54b5afc-e80a-4c14-aafd-8ff9ad7da046" containerName="maas-api" Nov 03 11:33:42.056592 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056585 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54b5afc-e80a-4c14-aafd-8ff9ad7da046" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056599 5086 cpu_manager.go:414] "containerMap: removing container" podUID="bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056604 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056618 5086 cpu_manager.go:414] "containerMap: removing container" podUID="6e657caa-0b2b-4318-ac6f-1d1e811992d5" containerName="payload-processing" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056624 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e657caa-0b2b-4318-ac6f-1d1e811992d5" containerName="payload-processing" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056633 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ea69424b-3f91-47a6-b534-604ca3d09332" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056638 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea69424b-3f91-47a6-b534-604ca3d09332" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056644 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c9670a7d-3aa9-4225-9cbb-783d6dcc4734" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056649 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9670a7d-3aa9-4225-9cbb-783d6dcc4734" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056654 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056661 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056669 5086 cpu_manager.go:414] "containerMap: removing container" podUID="1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056674 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056679 5086 cpu_manager.go:414] "containerMap: removing container" podUID="fd727856-e40a-4dc5-9c67-f44e8f89e123" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056684 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd727856-e40a-4dc5-9c67-f44e8f89e123" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056690 5086 cpu_manager.go:414] "containerMap: removing container" podUID="369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056695 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" containerName="maas-api" Nov 03 11:33:42.056692 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056703 5086 cpu_manager.go:414] "containerMap: removing container" podUID="d277c8b6-7753-4614-959d-e79d38a77fc4" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056708 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d277c8b6-7753-4614-959d-e79d38a77fc4" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056714 5086 cpu_manager.go:414] "containerMap: removing container" podUID="b4ea433f-d697-460c-8e67-32b03caa1440" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056718 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ea433f-d697-460c-8e67-32b03caa1440" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056728 5086 cpu_manager.go:414] "containerMap: removing container" podUID="390485cb-9990-4cb9-8a8e-38e57f80b2cf" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056733 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390485cb-9990-4cb9-8a8e-38e57f80b2cf" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056739 5086 cpu_manager.go:414] "containerMap: removing container" podUID="5f032621-ff28-4049-abe9-698a496a82f8" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056743 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f032621-ff28-4049-abe9-698a496a82f8" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056749 5086 cpu_manager.go:414] "containerMap: removing container" podUID="692a59fd-e1ee-402e-91f0-ff267e768e24" containerName="payload-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056753 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692a59fd-e1ee-402e-91f0-ff267e768e24" containerName="payload-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056759 5086 cpu_manager.go:414] "containerMap: removing container" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056764 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056770 5086 cpu_manager.go:414] "containerMap: removing container" podUID="482a443b-f6da-4644-a6ae-198e22602468" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056774 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482a443b-f6da-4644-a6ae-198e22602468" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056783 5086 cpu_manager.go:414] "containerMap: removing container" podUID="0f294516-cbba-4b13-87f4-e8a9815f82c8" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056787 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f294516-cbba-4b13-87f4-e8a9815f82c8" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056794 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerName="istio-proxy" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056798 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerName="istio-proxy" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056804 5086 cpu_manager.go:414] "containerMap: removing container" podUID="0063172a-cdd0-4c8d-b0c9-33f77c87c769" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056809 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0063172a-cdd0-4c8d-b0c9-33f77c87c769" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056817 5086 cpu_manager.go:414] "containerMap: removing container" podUID="955750bc-952d-4b29-b6e8-a4c119428fba" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056821 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955750bc-952d-4b29-b6e8-a4c119428fba" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056827 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056831 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056839 5086 cpu_manager.go:414] "containerMap: removing container" podUID="ac3318e5-f88a-40e6-9482-55a0e47c162e" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056843 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3318e5-f88a-40e6-9482-55a0e47c162e" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056849 5086 cpu_manager.go:414] "containerMap: removing container" podUID="9fcdd3bd-363b-4747-81c0-8bad0aee4a72" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056853 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcdd3bd-363b-4747-81c0-8bad0aee4a72" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056859 5086 cpu_manager.go:414] "containerMap: removing container" podUID="0602f904-1cbe-4088-9832-0961e0088df9" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056863 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0602f904-1cbe-4088-9832-0961e0088df9" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056872 5086 cpu_manager.go:414] "containerMap: removing container" podUID="7124d01f-392b-490a-9688-3a4e3b9f50f1" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056876 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7124d01f-392b-490a-9688-3a4e3b9f50f1" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056882 5086 cpu_manager.go:414] "containerMap: removing container" podUID="52ef7069-f15c-4d0e-89d3-2724ff2a0fb2" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056887 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ef7069-f15c-4d0e-89d3-2724ff2a0fb2" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056898 5086 cpu_manager.go:414] "containerMap: removing container" podUID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056902 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d41b16f-8d5a-476d-bdd1-b181795591da" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056908 5086 cpu_manager.go:414] "containerMap: removing container" podUID="c1228536-409d-41c3-aab5-b70ea3c1f4e6" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056913 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1228536-409d-41c3-aab5-b70ea3c1f4e6" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056923 5086 cpu_manager.go:414] "containerMap: removing container" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056927 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056935 5086 cpu_manager.go:414] "containerMap: removing container" podUID="40abd8ce-e351-4f5d-8bad-b774e6bd47c3" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056939 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40abd8ce-e351-4f5d-8bad-b774e6bd47c3" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056948 5086 cpu_manager.go:414] "containerMap: removing container" podUID="20fc1bd1-f950-46a7-8ec3-54555a42e1d0" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056952 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fc1bd1-f950-46a7-8ec3-54555a42e1d0" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056959 5086 cpu_manager.go:414] "containerMap: removing container" podUID="b70193f5-6efd-4009-a8a3-2d1405f4ce7c" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.056964 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70193f5-6efd-4009-a8a3-2d1405f4ce7c" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057027 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54b5afc-e80a-4c14-aafd-8ff9ad7da046" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057035 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8ec1bc-f67d-4d10-836c-7fa857ea9a91" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057043 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e657caa-0b2b-4318-ac6f-1d1e811992d5" containerName="payload-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057050 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea69424b-3f91-47a6-b534-604ca3d09332" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057055 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9670a7d-3aa9-4225-9cbb-783d6dcc4734" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057060 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8ec9b0-ec37-49ec-97ec-63c0a3ec50cc" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057066 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fab6dfc-dfe0-4ec8-bd5d-e59953b0290d" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057071 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd727856-e40a-4dc5-9c67-f44e8f89e123" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057077 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="369e9ba2-4a9d-4a48-9fa3-7a10ad04803a" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057082 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="d277c8b6-7753-4614-959d-e79d38a77fc4" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057087 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ea433f-d697-460c-8e67-32b03caa1440" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057094 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="390485cb-9990-4cb9-8a8e-38e57f80b2cf" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057098 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f032621-ff28-4049-abe9-698a496a82f8" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057102 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="692a59fd-e1ee-402e-91f0-ff267e768e24" containerName="payload-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057107 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="f053e8e0-87a9-49fc-ae65-5c07b771b89e" containerName="istio-proxy" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057118 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="482a443b-f6da-4644-a6ae-198e22602468" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057124 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f294516-cbba-4b13-87f4-e8a9815f82c8" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057130 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4973c07-e22e-4d8a-be06-505b9d3b7297" containerName="istio-proxy" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057135 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="0063172a-cdd0-4c8d-b0c9-33f77c87c769" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057141 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="955750bc-952d-4b29-b6e8-a4c119428fba" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057145 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf375e6-6ba5-4fca-b377-72060451aa71" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057150 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3318e5-f88a-40e6-9482-55a0e47c162e" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057154 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcdd3bd-363b-4747-81c0-8bad0aee4a72" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057159 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="0602f904-1cbe-4088-9832-0961e0088df9" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057164 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="7124d01f-392b-490a-9688-3a4e3b9f50f1" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057170 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ef7069-f15c-4d0e-89d3-2724ff2a0fb2" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057176 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1228536-409d-41c3-aab5-b70ea3c1f4e6" containerName="payload-pre-processing" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057183 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96bd9cc-e47f-40b3-ad46-2bd17d497592" containerName="istio-proxy" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057189 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="40abd8ce-e351-4f5d-8bad-b774e6bd47c3" containerName="main" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057195 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fc1bd1-f950-46a7-8ec3-54555a42e1d0" containerName="maas-api" Nov 03 11:33:42.057349 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.057200 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70193f5-6efd-4009-a8a3-2d1405f4ce7c" containerName="maas-api" Nov 03 11:33:42.061826 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.061808 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.066470 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.066445 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 11:33:42.066625 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.066602 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\"" Nov 03 11:33:42.068771 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.068736 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt"] Nov 03 11:33:42.081275 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.080810 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 11:33:42.081611 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.081571 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 11:33:42.148630 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.148599 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.148803 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.148636 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.148803 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.148660 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrp5\" (UniqueName: \"kubernetes.io/projected/29a969cc-b881-4b10-9c08-31bda1c1713b-kube-api-access-hbrp5\") pod \"maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.250335 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.250305 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.250533 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.250345 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.250533 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.250378 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrp5\" (UniqueName: \"kubernetes.io/projected/29a969cc-b881-4b10-9c08-31bda1c1713b-kube-api-access-hbrp5\") pod \"maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.252655 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.252625 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 11:33:42.252655 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.252632 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 11:33:42.257706 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.257681 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 11:33:42.261174 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.261148 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.262813 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.262790 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.268405 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.268382 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrp5\" (UniqueName: \"kubernetes.io/projected/29a969cc-b881-4b10-9c08-31bda1c1713b-kube-api-access-hbrp5\") pod \"maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.385441 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.385413 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\"" Nov 03 11:33:42.393579 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.393556 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:42.673214 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.673192 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt"] Nov 03 11:33:42.675179 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:33:42.675150 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a969cc_b881_4b10_9c08_31bda1c1713b.slice/crio-963f48534be9378b0660aede80da420813e2377044910c04a37ec854fb388800 WatchSource:0}: Error finding container 963f48534be9378b0660aede80da420813e2377044910c04a37ec854fb388800: Status 404 returned error can't find the container with id 963f48534be9378b0660aede80da420813e2377044910c04a37ec854fb388800 Nov 03 11:33:42.677556 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:42.677538 5086 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 11:33:43.263854 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:43.263826 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:33:43.271164 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:43.271134 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:33:43.291114 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:43.291088 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:33:43.296612 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:43.296593 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq8sx_27f4bb2a-dbbb-4cf1-96bc-e092ffd2efcf/ovn-acl-logging/0.log" Nov 03 11:33:43.637059 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:43.637021 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" event={"ID":"29a969cc-b881-4b10-9c08-31bda1c1713b","Type":"ContainerStarted","Data":"963f48534be9378b0660aede80da420813e2377044910c04a37ec854fb388800"} Nov 03 11:33:45.098539 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:45.098504 5086 scope.go:122] "RemoveContainer" containerID="813df2b8fe2109da4b91e3fe85e6eae71ec75bfc7884a16af7566475d70fde9d" Nov 03 11:33:46.655085 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:46.655053 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" event={"ID":"29a969cc-b881-4b10-9c08-31bda1c1713b","Type":"ContainerStarted","Data":"5b18eeb326c12790039b7a3d04dbc363ebf4830ff48bc75038bc2a437ac44f4a"} Nov 03 11:33:46.676185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:46.676126 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" podStartSLOduration=1.384851887 podStartE2EDuration="4.676111716s" totalImagesPullingTime="3.291259829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:33:46.67467678 +0000 UTC m=+3904.659548074" watchObservedRunningTime="2026-11-03 11:33:46.676111716 +0000 UTC m=+3904.660982993" Nov 03 11:33:46.891846 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:46.891811 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm"] Nov 03 11:33:47.662964 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:47.662927 5086 generic.go:356] "Generic (PLEG): container finished" podID="29a969cc-b881-4b10-9c08-31bda1c1713b" containerID="5b18eeb326c12790039b7a3d04dbc363ebf4830ff48bc75038bc2a437ac44f4a" exitCode=0 Nov 03 11:33:47.663442 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:47.662995 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" event={"ID":"29a969cc-b881-4b10-9c08-31bda1c1713b","Type":"ContainerDied","Data":"5b18eeb326c12790039b7a3d04dbc363ebf4830ff48bc75038bc2a437ac44f4a"} Nov 03 11:33:48.819784 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:48.819760 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:48.916125 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:48.916080 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-xks-ca\") pod \"29a969cc-b881-4b10-9c08-31bda1c1713b\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " Nov 03 11:33:48.916319 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:48.916148 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a969cc-b881-4b10-9c08-31bda1c1713b-kube-api-access-hbrp5\" (UniqueName: \"kubernetes.io/projected/29a969cc-b881-4b10-9c08-31bda1c1713b-kube-api-access-hbrp5\") pod \"29a969cc-b881-4b10-9c08-31bda1c1713b\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " Nov 03 11:33:48.916319 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:48.916184 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-openshift-ca\") pod \"29a969cc-b881-4b10-9c08-31bda1c1713b\" (UID: \"29a969cc-b881-4b10-9c08-31bda1c1713b\") " Nov 03 11:33:48.916649 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:48.916622 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-openshift-ca" pod "29a969cc-b881-4b10-9c08-31bda1c1713b" (UID: "29a969cc-b881-4b10-9c08-31bda1c1713b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:33:48.918255 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:48.918227 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-xks-ca" pod "29a969cc-b881-4b10-9c08-31bda1c1713b" (UID: "29a969cc-b881-4b10-9c08-31bda1c1713b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:33:48.918359 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:48.918278 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a969cc-b881-4b10-9c08-31bda1c1713b-kube-api-access-hbrp5" pod "29a969cc-b881-4b10-9c08-31bda1c1713b" (UID: "29a969cc-b881-4b10-9c08-31bda1c1713b"). InnerVolumeSpecName "kube-api-access-hbrp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:33:49.018111 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:49.018073 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:49.018320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:49.018114 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbrp5\" (UniqueName: \"kubernetes.io/projected/29a969cc-b881-4b10-9c08-31bda1c1713b-kube-api-access-hbrp5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:49.018320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:49.018132 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a969cc-b881-4b10-9c08-31bda1c1713b-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:49.675023 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:49.674994 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" Nov 03 11:33:49.675180 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:49.674992 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt" event={"ID":"29a969cc-b881-4b10-9c08-31bda1c1713b","Type":"ContainerDied","Data":"963f48534be9378b0660aede80da420813e2377044910c04a37ec854fb388800"} Nov 03 11:33:49.675180 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:49.675136 5086 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="963f48534be9378b0660aede80da420813e2377044910c04a37ec854fb388800" Nov 03 11:33:53.389359 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.389243 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z"] Nov 03 11:33:53.389710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.389696 5086 cpu_manager.go:414] "containerMap: removing container" podUID="29a969cc-b881-4b10-9c08-31bda1c1713b" containerName="revoke-keys" Nov 03 11:33:53.389710 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.389707 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a969cc-b881-4b10-9c08-31bda1c1713b" containerName="revoke-keys" Nov 03 11:33:53.389815 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.389804 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a969cc-b881-4b10-9c08-31bda1c1713b" containerName="revoke-keys" Nov 03 11:33:53.393228 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.393205 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.402084 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.402058 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 11:33:53.402327 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.402266 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 11:33:53.402428 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.402338 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 11:33:53.403083 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.403063 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\"" Nov 03 11:33:53.404766 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.404734 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z"] Nov 03 11:33:53.465051 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.465017 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hp66\" (UniqueName: \"kubernetes.io/projected/5b7e154d-282e-4843-9863-28adb1ba3285-kube-api-access-6hp66\") pod \"maas-api-revoke-sub-d0570c95-b8j2z\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.465209 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.465082 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0570c95-b8j2z\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.465341 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.465216 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0570c95-b8j2z\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.566781 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.566736 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hp66\" (UniqueName: \"kubernetes.io/projected/5b7e154d-282e-4843-9863-28adb1ba3285-kube-api-access-6hp66\") pod \"maas-api-revoke-sub-d0570c95-b8j2z\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.566977 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.566807 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0570c95-b8j2z\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.566977 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.566840 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0570c95-b8j2z\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.567497 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.567476 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0570c95-b8j2z\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.569185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.569164 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0570c95-b8j2z\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.574480 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.574458 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hp66\" (UniqueName: \"kubernetes.io/projected/5b7e154d-282e-4843-9863-28adb1ba3285-kube-api-access-6hp66\") pod \"maas-api-revoke-sub-d0570c95-b8j2z\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.707507 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.707426 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:53.984993 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:53.984962 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z"] Nov 03 11:33:53.987432 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:33:53.987404 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7e154d_282e_4843_9863_28adb1ba3285.slice/crio-3e1d01d61bcd33de721a79e5b278d8e8541a0222bb2eb32dc59351cb168a5a22 WatchSource:0}: Error finding container 3e1d01d61bcd33de721a79e5b278d8e8541a0222bb2eb32dc59351cb168a5a22: Status 404 returned error can't find the container with id 3e1d01d61bcd33de721a79e5b278d8e8541a0222bb2eb32dc59351cb168a5a22 Nov 03 11:33:54.273216 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:33:54.273139 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:33:54.703258 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:54.703221 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" event={"ID":"5b7e154d-282e-4843-9863-28adb1ba3285","Type":"ContainerStarted","Data":"16d6b7e9d9c6b93c85ddefe5a8122e3411ce453b7047e02a79dd8b96388753b0"} Nov 03 11:33:54.703670 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:54.703285 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" event={"ID":"5b7e154d-282e-4843-9863-28adb1ba3285","Type":"ContainerStarted","Data":"3e1d01d61bcd33de721a79e5b278d8e8541a0222bb2eb32dc59351cb168a5a22"} Nov 03 11:33:54.720108 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:54.720057 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" podStartSLOduration=1.720043887 podStartE2EDuration="1.720043887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:33:54.718844864 +0000 UTC m=+3912.703716147" watchObservedRunningTime="2026-11-03 11:33:54.720043887 +0000 UTC m=+3912.704915168" Nov 03 11:33:55.711762 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:55.711719 5086 generic.go:356] "Generic (PLEG): container finished" podID="5b7e154d-282e-4843-9863-28adb1ba3285" containerID="16d6b7e9d9c6b93c85ddefe5a8122e3411ce453b7047e02a79dd8b96388753b0" exitCode=0 Nov 03 11:33:55.712283 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:55.711795 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" event={"ID":"5b7e154d-282e-4843-9863-28adb1ba3285","Type":"ContainerDied","Data":"16d6b7e9d9c6b93c85ddefe5a8122e3411ce453b7047e02a79dd8b96388753b0"} Nov 03 11:33:56.877902 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:56.877876 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:57.002030 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.001933 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-openshift-ca\") pod \"5b7e154d-282e-4843-9863-28adb1ba3285\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " Nov 03 11:33:57.002030 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.001984 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-xks-ca\") pod \"5b7e154d-282e-4843-9863-28adb1ba3285\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " Nov 03 11:33:57.002030 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.002017 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7e154d-282e-4843-9863-28adb1ba3285-kube-api-access-6hp66\" (UniqueName: \"kubernetes.io/projected/5b7e154d-282e-4843-9863-28adb1ba3285-kube-api-access-6hp66\") pod \"5b7e154d-282e-4843-9863-28adb1ba3285\" (UID: \"5b7e154d-282e-4843-9863-28adb1ba3285\") " Nov 03 11:33:57.002465 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.002436 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-openshift-ca" pod "5b7e154d-282e-4843-9863-28adb1ba3285" (UID: "5b7e154d-282e-4843-9863-28adb1ba3285"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 11:33:57.004221 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.004190 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-xks-ca" pod "5b7e154d-282e-4843-9863-28adb1ba3285" (UID: "5b7e154d-282e-4843-9863-28adb1ba3285"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:33:57.004321 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.004258 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7e154d-282e-4843-9863-28adb1ba3285-kube-api-access-6hp66" pod "5b7e154d-282e-4843-9863-28adb1ba3285" (UID: "5b7e154d-282e-4843-9863-28adb1ba3285"). InnerVolumeSpecName "kube-api-access-6hp66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:33:57.103055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.103014 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:57.103055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.103046 5086 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7e154d-282e-4843-9863-28adb1ba3285-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:57.103055 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.103057 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hp66\" (UniqueName: \"kubernetes.io/projected/5b7e154d-282e-4843-9863-28adb1ba3285-kube-api-access-6hp66\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:33:57.723629 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.723599 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" Nov 03 11:33:57.723629 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.723608 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0570c95-b8j2z" event={"ID":"5b7e154d-282e-4843-9863-28adb1ba3285","Type":"ContainerDied","Data":"3e1d01d61bcd33de721a79e5b278d8e8541a0222bb2eb32dc59351cb168a5a22"} Nov 03 11:33:57.723847 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:33:57.723651 5086 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e1d01d61bcd33de721a79e5b278d8e8541a0222bb2eb32dc59351cb168a5a22" Nov 03 11:34:01.913476 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:01.913436 5086 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" containerID="cri-o://9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996" gracePeriod=45 Nov 03 11:34:02.062537 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.062432 5086 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 03 11:34:02.193553 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.193520 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 11:34:02.256775 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.256733 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e2a4eb-f888-4f51-8d56-a291557109d4-kube-api-access-mpx8q\" (UniqueName: \"kubernetes.io/projected/62e2a4eb-f888-4f51-8d56-a291557109d4-kube-api-access-mpx8q\") pod \"62e2a4eb-f888-4f51-8d56-a291557109d4\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " Nov 03 11:34:02.256975 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.256798 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-tmp-dir\") pod \"62e2a4eb-f888-4f51-8d56-a291557109d4\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " Nov 03 11:34:02.256975 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.256875 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-home\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-home\") pod \"62e2a4eb-f888-4f51-8d56-a291557109d4\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " Nov 03 11:34:02.256975 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.256908 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-model-cache\") pod \"62e2a4eb-f888-4f51-8d56-a291557109d4\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " Nov 03 11:34:02.256975 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.256965 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-dshm\") pod \"62e2a4eb-f888-4f51-8d56-a291557109d4\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " Nov 03 11:34:02.257235 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.257004 5086 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e2a4eb-f888-4f51-8d56-a291557109d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/62e2a4eb-f888-4f51-8d56-a291557109d4-tls-certs\") pod \"62e2a4eb-f888-4f51-8d56-a291557109d4\" (UID: \"62e2a4eb-f888-4f51-8d56-a291557109d4\") " Nov 03 11:34:02.257235 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.257050 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-tmp-dir" pod "62e2a4eb-f888-4f51-8d56-a291557109d4" (UID: "62e2a4eb-f888-4f51-8d56-a291557109d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:34:02.257235 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.257104 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-home" pod "62e2a4eb-f888-4f51-8d56-a291557109d4" (UID: "62e2a4eb-f888-4f51-8d56-a291557109d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:34:02.257445 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.257280 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-model-cache" pod "62e2a4eb-f888-4f51-8d56-a291557109d4" (UID: "62e2a4eb-f888-4f51-8d56-a291557109d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:34:02.257530 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.257495 5086 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:34:02.257530 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.257518 5086 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:34:02.257649 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.257530 5086 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:34:02.259508 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.259477 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e2a4eb-f888-4f51-8d56-a291557109d4-kube-api-access-mpx8q" pod "62e2a4eb-f888-4f51-8d56-a291557109d4" (UID: "62e2a4eb-f888-4f51-8d56-a291557109d4"). InnerVolumeSpecName "kube-api-access-mpx8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 11:34:02.259627 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.259556 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-dshm" pod "62e2a4eb-f888-4f51-8d56-a291557109d4" (UID: "62e2a4eb-f888-4f51-8d56-a291557109d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 11:34:02.259627 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.259610 5086 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e2a4eb-f888-4f51-8d56-a291557109d4-tls-certs" pod "62e2a4eb-f888-4f51-8d56-a291557109d4" (UID: "62e2a4eb-f888-4f51-8d56-a291557109d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 11:34:02.359042 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.359007 5086 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e2a4eb-f888-4f51-8d56-a291557109d4-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:34:02.359042 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.359039 5086 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e2a4eb-f888-4f51-8d56-a291557109d4-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:34:02.359263 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.359100 5086 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpx8q\" (UniqueName: \"kubernetes.io/projected/62e2a4eb-f888-4f51-8d56-a291557109d4-kube-api-access-mpx8q\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 11:34:02.751248 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.751212 5086 generic.go:356] "Generic (PLEG): container finished" podID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerID="9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996" exitCode=0 Nov 03 11:34:02.751451 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.751255 5086 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" Nov 03 11:34:02.751451 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.751315 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" event={"ID":"62e2a4eb-f888-4f51-8d56-a291557109d4","Type":"ContainerDied","Data":"9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996"} Nov 03 11:34:02.751451 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.751369 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm" event={"ID":"62e2a4eb-f888-4f51-8d56-a291557109d4","Type":"ContainerDied","Data":"7ba3a48220cb08ef346d5a487ad04e87c962315afdd6b9dfb16f787c63975edd"} Nov 03 11:34:02.751451 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.751393 5086 scope.go:122] "RemoveContainer" containerID="9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996" Nov 03 11:34:02.772257 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.772238 5086 scope.go:122] "RemoveContainer" containerID="9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996" Nov 03 11:34:02.772561 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:34:02.772542 5086 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996\": container with ID starting with 9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996 not found: ID does not exist" containerID="9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996" Nov 03 11:34:02.772615 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.772571 5086 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996"} err="failed to get container status \"9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996\": rpc error: code = NotFound desc = could not find container \"9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996\": container with ID starting with 9df84d7887c42ec4b128f0f2421307214e121bc05f26a27d3898952c0d628996 not found: ID does not exist" Nov 03 11:34:02.782652 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.782630 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm"] Nov 03 11:34:02.786680 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:02.786658 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/simulator-w3-b1c6f6-kserve-755c4cc6f4-wc2dm"] Nov 03 11:34:03.610286 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:03.610248 5086 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt"] Nov 03 11:34:03.614467 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:03.614438 5086 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b1c6f6-p5pnt"] Nov 03 11:34:04.169617 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:04.169588 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a969cc-b881-4b10-9c08-31bda1c1713b" path="/var/lib/kubelet/pods/29a969cc-b881-4b10-9c08-31bda1c1713b/volumes" Nov 03 11:34:04.170013 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:04.170001 5086 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" path="/var/lib/kubelet/pods/62e2a4eb-f888-4f51-8d56-a291557109d4/volumes" Nov 03 11:34:11.593223 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:11.593180 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b56ffbcf8-d4flt_ac3c3232-9196-41f4-88ab-ffa55948e449/authorino/0.log" Nov 03 11:34:11.806049 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:11.806022 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k95vx_09b8b602-a32b-4d2e-ae58-d4d15c56457e/manager/0.log" Nov 03 11:34:13.012153 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:13.012113 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4shtg_578a0ec5-eda4-4576-acd8-8cd12251d4dd/manager/0.log" Nov 03 11:34:13.831391 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:13.831357 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b675dd85b-h27pj_be5179f0-1851-42f5-8429-8ffa5892af05/manager/0.log" Nov 03 11:34:14.231011 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:14.230935 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-c2657_f779485f-1f41-481a-9fb8-4ea051dfa92f/manager/0.log" Nov 03 11:34:15.110334 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:15.110282 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kjdj9_d804b100-c032-4251-8baf-c40f1e01f919/server/0.log" Nov 03 11:34:15.303188 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:15.303158 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-8lglj_1af95fda-8dff-495e-8d59-334e7aeca7f8/manager/0.log" Nov 03 11:34:15.528185 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:15.528084 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9pz5c_ad41c321-4063-400c-9415-2574d48dcc5b/proxy/0.log" Nov 03 11:34:16.444126 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:16.444087 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b56ffbcf8-d4flt_ac3c3232-9196-41f4-88ab-ffa55948e449/authorino/0.log" Nov 03 11:34:24.273884 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:34:24.273838 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:34:47.699870 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:47.699837 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b675dd85b-h27pj_be5179f0-1851-42f5-8429-8ffa5892af05/manager/0.log" Nov 03 11:34:48.096535 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:48.096503 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-c2657_f779485f-1f41-481a-9fb8-4ea051dfa92f/manager/0.log" Nov 03 11:34:49.004696 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:49.004669 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kjdj9_d804b100-c032-4251-8baf-c40f1e01f919/server/0.log" Nov 03 11:34:49.200335 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:49.200279 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-8lglj_1af95fda-8dff-495e-8d59-334e7aeca7f8/manager/0.log" Nov 03 11:34:49.428737 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:49.428710 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9pz5c_ad41c321-4063-400c-9415-2574d48dcc5b/proxy/0.log" Nov 03 11:34:52.121222 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:52.121191 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d0570c95-b8j2z_5b7e154d-282e-4843-9863-28adb1ba3285/revoke-keys/0.log" Nov 03 11:34:54.273347 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:34:54.273283 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:34:54.747664 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:54.747633 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b56ffbcf8-d4flt_ac3c3232-9196-41f4-88ab-ffa55948e449/authorino/0.log" Nov 03 11:34:54.942828 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:54.942800 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k95vx_09b8b602-a32b-4d2e-ae58-d4d15c56457e/manager/0.log" Nov 03 11:34:56.078682 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:56.078646 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4shtg_578a0ec5-eda4-4576-acd8-8cd12251d4dd/manager/0.log" Nov 03 11:34:57.050127 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:57.050095 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9mtdw_b4ec4160-f315-4999-bdb3-a24364695cff/discovery/0.log" Nov 03 11:34:57.852186 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:57.852154 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-5p4bj_b7804ed8-ba49-4936-99f0-f02253369475/istio-proxy/0.log" Nov 03 11:34:58.814464 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:58.814435 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-lrmsr_99bbba2f-874c-4246-9ec3-e92e117a4159/istio-proxy/0.log" Nov 03 11:34:58.985320 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:58.985273 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-n7vfj_bb8055e0-8ed8-4f82-9816-2afed09b243e/istio-proxy/0.log" Nov 03 11:34:59.756229 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:34:59.756198 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-788447d7cc-tl5jd_0dbdd6e7-a8de-4990-9aec-358480af8521/router/0.log" Nov 03 11:35:00.756523 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:00.756496 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-s7nj5_bb46a330-9188-4da0-9d44-b2795c71995e/main/0.log" Nov 03 11:35:01.111864 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:01.111832 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-bpt97_266331b8-afce-492e-830c-933449e1a765/main/0.log" Nov 03 11:35:01.480222 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:01.480140 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vtck4_70328620-3e3d-4bd3-a573-80e8117c8d64/main/0.log" Nov 03 11:35:01.836762 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:01.836730 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-h8g8r_a5af7ba9-03fa-4655-b7bd-3f18d457be7b/main/0.log" Nov 03 11:35:24.272892 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:35:24.272810 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:35:26.796981 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:26.796943 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4vttz_ed900688-4264-4ba8-8daa-83e717a6f13f/global-pull-secret-syncer/0.log" Nov 03 11:35:26.926971 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:26.926945 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-q22k4_69dd95c4-b504-46d3-97a9-ac864aef3f85/connection-checker/0.log" Nov 03 11:35:26.953502 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:26.953474 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qkwqb_1dd4ba4e-edba-4732-9f72-54c83b78d6fc/connection-checker/0.log" Nov 03 11:35:26.979152 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:26.979123 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wk475_3e7c1aa6-a7c5-4ed6-a871-43a9bf568693/connection-checker/0.log" Nov 03 11:35:27.192719 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:27.192686 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x9pz8_9bac6548-3f94-4861-84ef-5bf4bfd57409/konnectivity-agent/0.log" Nov 03 11:35:27.293810 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:27.293777 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal_bb77e2dfdbc4a16f3bcede022ac475a7/haproxy/0.log" Nov 03 11:35:34.627744 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:34.627713 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b56ffbcf8-d4flt_ac3c3232-9196-41f4-88ab-ffa55948e449/authorino/0.log" Nov 03 11:35:34.663010 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:34.662985 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k95vx_09b8b602-a32b-4d2e-ae58-d4d15c56457e/manager/0.log" Nov 03 11:35:36.217769 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:36.217737 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4shtg_578a0ec5-eda4-4576-acd8-8cd12251d4dd/manager/0.log" Nov 03 11:35:37.700178 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:37.700146 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-rchbs_5a9404d9-1240-451f-8064-255583bb5da6/cluster-monitoring-operator/0.log" Nov 03 11:35:38.021629 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.021558 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wr9wd_625af666-e1d5-4bd0-b747-84cbae4aee5a/node-exporter/0.log" Nov 03 11:35:38.056754 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.049852 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wr9wd_625af666-e1d5-4bd0-b747-84cbae4aee5a/kube-rbac-proxy/0.log" Nov 03 11:35:38.075438 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.075412 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wr9wd_625af666-e1d5-4bd0-b747-84cbae4aee5a/init-textfile/0.log" Nov 03 11:35:38.092033 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.092006 5086 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj"] Nov 03 11:35:38.092468 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.092454 5086 cpu_manager.go:414] "containerMap: removing container" podUID="5b7e154d-282e-4843-9863-28adb1ba3285" containerName="revoke-keys" Nov 03 11:35:38.092521 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.092469 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7e154d-282e-4843-9863-28adb1ba3285" containerName="revoke-keys" Nov 03 11:35:38.092521 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.092479 5086 cpu_manager.go:414] "containerMap: removing container" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" Nov 03 11:35:38.092521 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.092484 5086 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" Nov 03 11:35:38.092622 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.092576 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e2a4eb-f888-4f51-8d56-a291557109d4" containerName="main" Nov 03 11:35:38.092622 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.092593 5086 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7e154d-282e-4843-9863-28adb1ba3285" containerName="revoke-keys" Nov 03 11:35:38.096127 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.096101 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.104753 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.104729 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj"] Nov 03 11:35:38.186772 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.186732 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-proc\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.186943 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.186792 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-podres\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.186943 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.186822 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8qb\" (UniqueName: \"kubernetes.io/projected/a1e57ee2-b1fb-4d82-ac11-148169de1895-kube-api-access-6q8qb\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.186943 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.186848 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-lib-modules\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.187067 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.187013 5086 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-sys\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.195148 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.195125 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v48x6\"/\"openshift-service-ca.crt\"" Nov 03 11:35:38.195280 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.195269 5086 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v48x6\"/\"kube-root-ca.crt\"" Nov 03 11:35:38.195368 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.195351 5086 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v48x6\"/\"default-dockercfg-25c4q\"" Nov 03 11:35:38.287737 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.287658 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-podres\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.287737 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.287694 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8qb\" (UniqueName: \"kubernetes.io/projected/a1e57ee2-b1fb-4d82-ac11-148169de1895-kube-api-access-6q8qb\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.287737 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.287715 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-lib-modules\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.288006 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.287780 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-sys\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.288006 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.287795 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-podres\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.288006 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.287859 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-sys\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.288006 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.287861 5086 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-proc\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.288006 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.287874 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-lib-modules\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.288006 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.287903 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1e57ee2-b1fb-4d82-ac11-148169de1895-proc\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.295022 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.294999 5086 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8qb\" (UniqueName: \"kubernetes.io/projected/a1e57ee2-b1fb-4d82-ac11-148169de1895-kube-api-access-6q8qb\") pod \"perf-node-gather-daemonset-6xpzj\" (UID: \"a1e57ee2-b1fb-4d82-ac11-148169de1895\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.412473 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.412425 5086 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:38.416559 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.416536 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xmlk2_4df3ca90-d4ac-4e97-9056-76645061f149/prometheus-operator/0.log" Nov 03 11:35:38.434106 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.434085 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xmlk2_4df3ca90-d4ac-4e97-9056-76645061f149/kube-rbac-proxy/0.log" Nov 03 11:35:38.460525 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.460498 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qp2wn_cb353b40-b1b4-43cf-b7a2-9d1fa9d3f225/prometheus-operator-admission-webhook/0.log" Nov 03 11:35:38.685936 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:38.685910 5086 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj"] Nov 03 11:35:38.688036 ip-10-0-139-90 kubenswrapper[5086]: W1103 11:35:38.688006 5086 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda1e57ee2_b1fb_4d82_ac11_148169de1895.slice/crio-88ac3ecf6c521bbecf3351ee01bf86f751c5a18e08384ff7071a6706181a967d WatchSource:0}: Error finding container 88ac3ecf6c521bbecf3351ee01bf86f751c5a18e08384ff7071a6706181a967d: Status 404 returned error can't find the container with id 88ac3ecf6c521bbecf3351ee01bf86f751c5a18e08384ff7071a6706181a967d Nov 03 11:35:39.272189 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:39.272150 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" event={"ID":"a1e57ee2-b1fb-4d82-ac11-148169de1895","Type":"ContainerStarted","Data":"662411b53eafa14414362aa65eefa9e4c7f0b011c2450017edfb7f90e49a8be8"} Nov 03 11:35:39.272610 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:39.272207 5086 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" event={"ID":"a1e57ee2-b1fb-4d82-ac11-148169de1895","Type":"ContainerStarted","Data":"88ac3ecf6c521bbecf3351ee01bf86f751c5a18e08384ff7071a6706181a967d"} Nov 03 11:35:39.272610 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:39.272238 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:39.288844 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:39.288797 5086 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" podStartSLOduration=1.288784937 podStartE2EDuration="1.288784937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:35:39.288081696 +0000 UTC m=+4017.272952962" watchObservedRunningTime="2026-11-03 11:35:39.288784937 +0000 UTC m=+4017.273656219" Nov 03 11:35:40.017260 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:40.017231 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-h65vg_cb00dfc2-12f0-4fc3-ba37-0bc83b344602/networking-console-plugin/0.log" Nov 03 11:35:40.712373 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:40.712340 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/1.log" Nov 03 11:35:40.717826 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:40.717798 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2m2xr_aedffb88-1606-439b-968d-ab12c5a2b984/console-operator/2.log" Nov 03 11:35:41.314584 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:41.314555 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-7plpl_44bd20f9-05bf-4ca7-9da3-4fd87c311c2c/download-server/0.log" Nov 03 11:35:41.640116 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:41.640085 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b675dd85b-h27pj_be5179f0-1851-42f5-8429-8ffa5892af05/manager/0.log" Nov 03 11:35:41.709905 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:41.709875 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-c2657_f779485f-1f41-481a-9fb8-4ea051dfa92f/manager/0.log" Nov 03 11:35:42.025766 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:42.025692 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-h8sdw_b86c6b47-2b14-4151-a04a-12c1505e928a/volume-data-source-validator/0.log" Nov 03 11:35:42.859255 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:42.859229 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kjdj9_d804b100-c032-4251-8baf-c40f1e01f919/server/0.log" Nov 03 11:35:43.582979 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:43.582948 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fb26v_f8ea16e9-1545-4881-b7c2-883d8b09a6fe/dns/0.log" Nov 03 11:35:43.613942 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:43.613917 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fb26v_f8ea16e9-1545-4881-b7c2-883d8b09a6fe/kube-rbac-proxy/0.log" Nov 03 11:35:43.765120 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:43.765087 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jhp98_35d6f6db-753a-4d1e-ae56-58659a5baab1/dns-node-resolver/0.log" Nov 03 11:35:44.147992 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:44.147960 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-8lglj_1af95fda-8dff-495e-8d59-334e7aeca7f8/manager/0.log" Nov 03 11:35:44.176039 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:44.176012 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9pz5c_ad41c321-4063-400c-9415-2574d48dcc5b/proxy/0.log" Nov 03 11:35:44.579920 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:44.579865 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65b85b84d7-v8cxp_e00ff8a7-26cf-4089-a4da-7ddb27de2a75/registry/0.log" Nov 03 11:35:44.666148 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:44.666117 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ssnbk_35dabb24-0583-4de3-9a5d-21c8ef04c7c5/node-ca/0.log" Nov 03 11:35:45.289486 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:45.289460 5086 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-6xpzj" Nov 03 11:35:45.305840 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:45.305805 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b675dd85b-h27pj_be5179f0-1851-42f5-8429-8ffa5892af05/manager/0.log" Nov 03 11:35:45.378221 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:45.378196 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-c2657_f779485f-1f41-481a-9fb8-4ea051dfa92f/manager/0.log" Nov 03 11:35:46.617759 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:46.617730 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kjdj9_d804b100-c032-4251-8baf-c40f1e01f919/server/0.log" Nov 03 11:35:48.352447 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:48.352419 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9mtdw_b4ec4160-f315-4999-bdb3-a24364695cff/discovery/0.log" Nov 03 11:35:48.461721 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:48.461691 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-8lglj_1af95fda-8dff-495e-8d59-334e7aeca7f8/manager/0.log" Nov 03 11:35:48.575625 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:48.575593 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9pz5c_ad41c321-4063-400c-9415-2574d48dcc5b/proxy/0.log" Nov 03 11:35:50.052251 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:50.052219 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b675dd85b-h27pj_be5179f0-1851-42f5-8429-8ffa5892af05/manager/0.log" Nov 03 11:35:50.131375 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:50.131349 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-c2657_f779485f-1f41-481a-9fb8-4ea051dfa92f/manager/0.log" Nov 03 11:35:51.404135 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:51.404100 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-5p4bj_b7804ed8-ba49-4936-99f0-f02253369475/istio-proxy/0.log" Nov 03 11:35:51.616981 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:51.616930 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-lrmsr_99bbba2f-874c-4246-9ec3-e92e117a4159/istio-proxy/0.log" Nov 03 11:35:51.646935 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:51.646911 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-n7vfj_bb8055e0-8ed8-4f82-9816-2afed09b243e/istio-proxy/0.log" Nov 03 11:35:51.678597 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:51.678533 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kjdj9_d804b100-c032-4251-8baf-c40f1e01f919/server/0.log" Nov 03 11:35:51.788397 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:51.788371 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-788447d7cc-tl5jd_0dbdd6e7-a8de-4990-9aec-358480af8521/router/0.log" Nov 03 11:35:53.191431 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:53.191397 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9mtdw_b4ec4160-f315-4999-bdb3-a24364695cff/discovery/0.log" Nov 03 11:35:53.514467 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:53.514380 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-8lglj_1af95fda-8dff-495e-8d59-334e7aeca7f8/manager/0.log" Nov 03 11:35:53.607159 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:53.607130 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9pz5c_ad41c321-4063-400c-9415-2574d48dcc5b/proxy/0.log" Nov 03 11:35:54.278936 ip-10-0-139-90 kubenswrapper[5086]: E1103 11:35:54.278892 5086 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rhvqw" containerName="extractor" Nov 03 11:35:54.779105 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:54.779076 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhr4s_4a3c3eab-aa4f-4e5b-8b91-1cea0bf245d8/serve-healthcheck-canary/0.log" Nov 03 11:35:54.952967 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:54.952933 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-5p4bj_b7804ed8-ba49-4936-99f0-f02253369475/istio-proxy/0.log" Nov 03 11:35:55.053034 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:55.052957 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-lrmsr_99bbba2f-874c-4246-9ec3-e92e117a4159/istio-proxy/0.log" Nov 03 11:35:55.080355 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:55.080327 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-n7vfj_bb8055e0-8ed8-4f82-9816-2afed09b243e/istio-proxy/0.log" Nov 03 11:35:55.168922 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:55.168893 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-788447d7cc-tl5jd_0dbdd6e7-a8de-4990-9aec-358480af8521/router/0.log" Nov 03 11:35:55.483869 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:55.483841 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-8rp5q_5f7b33fa-69cf-4582-a4b4-844ed91e2e26/insights-operator/0.log" Nov 03 11:35:55.484841 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:55.484821 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-8rp5q_5f7b33fa-69cf-4582-a4b4-844ed91e2e26/insights-operator/1.log" Nov 03 11:35:55.674949 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:55.674915 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rhvqw_0945eba8-6898-4dfe-ae35-687496c3b596/kube-rbac-proxy/0.log" Nov 03 11:35:55.703387 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:55.703362 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rhvqw_0945eba8-6898-4dfe-ae35-687496c3b596/exporter/0.log" Nov 03 11:35:55.729015 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:55.728987 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rhvqw_0945eba8-6898-4dfe-ae35-687496c3b596/extractor/0.log" Nov 03 11:35:55.779350 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:55.779263 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-f74wv-nz4fj_e5a3a09a-d7cf-4f67-9b22-99456a86976b/insights-gathering/0.log" Nov 03 11:35:56.584729 ip-10-0-139-90 kubenswrapper[5086]: I1103 11:35:56.584698 5086 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9mtdw_b4ec4160-f315-4999-bdb3-a24364695cff/discovery/0.log"